Large hash with mixed symbol and string key causing segfaults with to_json

[Jiig]

In ruby 3.4.6 and 4.0.0 we are seeing strange exceptions and even segfaults when trying to dump JSON strings of large hashes

Hash example:

test_data = {
  "source" => "source",
  "startTime" => Time.now.utc.iso8601,
  "endTime" => Time.now.utc.iso8601,
  "createTime" => Time.now.utc.iso8601,
  "dataType" => "foo",
  "uuid" => SecureRandom.uuid,
  "flag" => true,
  "data" => [[]...] # Large array of arrays, sub arrays contain date string and lots of floats
}

I've traced the root cause in our code to setting a key in the hash to a symbol version later in the code:

test_data[:isTest] = true

I've fixed this in our code by just using a string key (as we should have been).
However the segfaults are concerning, and the behavior of the exceptions also doesn't make much sense to me.

When I call to_json on the hash I get one of two exceptions:

'JSON::Ext::Generator::GeneratorMethods::Hash#to_json': no implicit conversion of Integer into String (TypeError)

Or sometimes its a float:

'JSON::Ext::Generator::GeneratorMethods::Hash#to_json': no implicit conversion of Float into String (TypeError)

And then sometimes it segfaults and crashes ruby (see link for crash dump):

https://gist.github.com/Jiig/b89f887c9c601345afeade76dfffd2a7

I'm not sure if its the layout of these hashes in tandem with the mixed keys, or if using mixed keys at all in larges hashes is causing the issue.

And if the answer is "don't use mixed keys" I understand, just wanted to raise the fact that its crashing ruby.

Reproducible ruby code:

require 'json'
require 'securerandom'

test_data = {
  "source" => "source",
  "startTime" => Time.now.utc.iso8601,
  "endTime" => Time.now.utc.iso8601,
  "createTime" => Time.now.utc.iso8601,
  "dataType" => "foo",
  "uuid" => SecureRandom.uuid,
  "flag" => true,
  "data" => []
}

# Changing the amount of data points changes the behavior
# small amounts like 100 have no issue, but once you start getting over 300 or so it starts having issues (didn't find an exact number)
(1..10000).each do |n|
  test_data["data"] << [
    Time.now.utc.iso8601,
    Random.rand(-100000.0..100000.0),
    Random.rand(-100000.0..100000.0),
    Random.rand(-100000.0..100000.0),
    Random.rand(-100000.0..100000.0),
    Random.rand(-100000.0..100000.0),
    Random.rand(-100000.0..100000.0),
    1.0,
    0.0,
    0.0,
    1.0,
    0.0,
    1.0
  ]
end
# Convert the random data to JSON, doesn't have any issues as expected
test_data.to_json

# Then if we set the symbol key:
test_data[:flag] = false

# The to_json method starts raising exceptions or segfaults
# Worth noting that the JSON.pretty_generate_method does not have the same issue
# JSON.pretty_generate(test_data) # If you replace the to_json below with this it doesn't have any issues
test_data.to_json

[byroot]

Thank you for the report, however I'm afraid it's not failing at all on my machine. I see from your crash report you are using json 2.18.0, which I also tried with the exact same ruby 3.4.6.

Does it reproduce with just this script on your machine? Or is it inside a larger application?

[byroot]

Or is it inside a larger application?

Nevermind, i can see from the crash report that you have nothing more than just the script.

[Jiig]

Yeah its just using this script, I can give more system details if that would be helpful

[byroot]

Alright, I'm able to repro from a Linux container, seem to be linux only. I'll keep digging.

[byroot]

So this is a memory corruption bug. Somehow, one of the hash keys end up entirely corrupted, and this cause Ruby to crash in rb_string_value_ptr() (routine to convert any value into string).

(gdb) bt 20
#0  0x0000ffffac1d0514 in hash_table_index (tbl=0xffff90595ca0, key=156) at /usr/src/ruby/id_table.c:136
#1  rb_id_table_lookup (tbl=tbl@entry=0xffff90595ca0, id=id@entry=156, valp=valp@entry=0xfffff3bc6640) at /usr/src/ruby/id_table.c:230
#2  0x0000ffffac222198 in cached_callable_method_entry (klass=187651314773968, mid=156) at /usr/src/ruby/vm_method.c:1427
#3  callable_method_entry_or_negative (klass=klass@entry=187651314773968, mid=mid@entry=156, defined_class_ptr=defined_class_ptr@entry=0xfffff3bc66b8)
    at /usr/src/ruby/vm_method.c:1497
#4  0x0000ffffac22d3d4 in callable_method_entry (klass=187651314773968, mid=156, defined_class_ptr=0xfffff3bc66b8) at /usr/src/ruby/vm_method.c:1534
#5  vm_respond_to (ec=ec@entry=0xaaaaf97a04c0, klass=klass@entry=187651314773968, obj=obj@entry=281473564412656, id=id@entry=3265, priv=priv@entry=1)
    at /usr/src/ruby/vm_method.c:2898
#6  0x0000ffffac22d858 in check_funcall_respond_to (ec=0xaaaaf97a04c0, klass=187651314773968, recv=281473564412656, mid=3265) at /usr/src/ruby/vm_eval.c:607
#7  rb_check_funcall_default_kw (recv=281473564412656, mid=3265, argc=0, argv=0x0, def=36, kw_splat=0) at /usr/src/ruby/vm_eval.c:681
#8  rb_check_funcall (recv=recv@entry=281473564412656, mid=mid@entry=3265, argc=argc@entry=0, argv=argv@entry=0x0) at /usr/src/ruby/vm_eval.c:670
#9  0x0000ffffac0e0408 in convert_type_with_id (val=281473564412656, tname=0xffffac3ddd60 "String", method=3265, raise=1, index=-1) at object.c:3137
#10 rb_convert_type_with_id (val=281473564412656, type=5, tname=0xffffac3ddd60 "String", method=3265) at object.c:3199
#11 rb_convert_type_with_id (val=281473564412656, type=type@entry=5, tname=tname@entry=0xffffac3ddd60 "String", method=method@entry=3265) at object.c:3194
#12 0x0000ffffac1a7df4 in rb_str_to_str (str=<optimized out>) at string.c:1711
#13 0x0000ffffac1accd0 in rb_string_value (ptr=ptr@entry=0xfffff3bc69c0) at string.c:2720
#14 0x0000ffffac1b1d7c in rb_string_value_ptr (ptr=ptr@entry=0xfffff3bc69c0) at string.c:2729
#15 0x0000ffff902a78d4 in fbuffer_append_str (fb=0xfffff3bc6ea0, str=<optimized out>) at ../../../../../../ext/json/ext/generator/../fbuffer/fbuffer.h:164
#16 json_object_i (key=<optimized out>, val=0, _arg=<optimized out>) at ../../../../../../ext/json/ext/generator/generator.c:1254
#17 0x0000ffffac069414 in hash_ar_foreach_iter (key=<optimized out>, value=<optimized out>, argp=281474770955264, error=0) at hash.c:1302
#18 ar_foreach_check (func=<optimized out>, never=36, hash=<optimized out>, arg=281474770955264) at hash.c:944
#19 hash_foreach_call (arg=281474770955264) at hash.c:1437
(More stack frames follow...)
(gdb) bt 40
#0  0x0000ffffac1d0514 in hash_table_index (tbl=0xffff90595ca0, key=156) at /usr/src/ruby/id_table.c:136
#1  rb_id_table_lookup (tbl=tbl@entry=0xffff90595ca0, id=id@entry=156, valp=valp@entry=0xfffff3bc6640) at /usr/src/ruby/id_table.c:230
#2  0x0000ffffac222198 in cached_callable_method_entry (klass=187651314773968, mid=156) at /usr/src/ruby/vm_method.c:1427
#3  callable_method_entry_or_negative (klass=klass@entry=187651314773968, mid=mid@entry=156, defined_class_ptr=defined_class_ptr@entry=0xfffff3bc66b8)
    at /usr/src/ruby/vm_method.c:1497
#4  0x0000ffffac22d3d4 in callable_method_entry (klass=187651314773968, mid=156, defined_class_ptr=0xfffff3bc66b8) at /usr/src/ruby/vm_method.c:1534
#5  vm_respond_to (ec=ec@entry=0xaaaaf97a04c0, klass=klass@entry=187651314773968, obj=obj@entry=281473564412656, id=id@entry=3265, priv=priv@entry=1)
    at /usr/src/ruby/vm_method.c:2898
#6  0x0000ffffac22d858 in check_funcall_respond_to (ec=0xaaaaf97a04c0, klass=187651314773968, recv=281473564412656, mid=3265) at /usr/src/ruby/vm_eval.c:607
#7  rb_check_funcall_default_kw (recv=281473564412656, mid=3265, argc=0, argv=0x0, def=36, kw_splat=0) at /usr/src/ruby/vm_eval.c:681
#8  rb_check_funcall (recv=recv@entry=281473564412656, mid=mid@entry=3265, argc=argc@entry=0, argv=argv@entry=0x0) at /usr/src/ruby/vm_eval.c:670
#9  0x0000ffffac0e0408 in convert_type_with_id (val=281473564412656, tname=0xffffac3ddd60 "String", method=3265, raise=1, index=-1) at object.c:3137
#10 rb_convert_type_with_id (val=281473564412656, type=5, tname=0xffffac3ddd60 "String", method=3265) at object.c:3199
#11 rb_convert_type_with_id (val=281473564412656, type=type@entry=5, tname=tname@entry=0xffffac3ddd60 "String", method=method@entry=3265) at object.c:3194
#12 0x0000ffffac1a7df4 in rb_str_to_str (str=<optimized out>) at string.c:1711
#13 0x0000ffffac1accd0 in rb_string_value (ptr=ptr@entry=0xfffff3bc69c0) at string.c:2720
#14 0x0000ffffac1b1d7c in rb_string_value_ptr (ptr=ptr@entry=0xfffff3bc69c0) at string.c:2729
#15 0x0000ffff902a78d4 in fbuffer_append_str (fb=0xfffff3bc6ea0, str=<optimized out>) at ../../../../../../ext/json/ext/generator/../fbuffer/fbuffer.h:164
#16 json_object_i (key=<optimized out>, val=0, _arg=<optimized out>) at ../../../../../../ext/json/ext/generator/generator.c:1254

I checked the type of the key we're passing:

(gdb) p/x ((struct RBasic *)ptr)->flags & RUBY_T_MASK
$9 = 0x10

Which is garbage, there is no such type in Ruby: https://github.com/ruby/ruby/blob/997943219c2d44e60d1d7ae25c8f78c99ce6d970/include/ruby/internal/value_type.h#L131

So the only explanation is that either we're pointing at garbage, or either we corrupted the original key objects.

What is interesting is that it seems the corruption is triggered when json generate the warning for the duplicate key situation:

json/lib/json/common.rb

Line 191 in 062fcdd

message = "detected duplicate key #{key_str.inspect} in #{hash.inspect}"

If I remove #{hash.inspect} from the deprecation message, it no longer seem to fail. But I need to dig deeper.

And if the answer is "don't use mixed keys" I understand, just wanted to raise the fact that its crashing ruby.

Yeah, to be clear these duplicated keys thing is deprecated and will raise an error in the future. See: https://github.com/ruby/json/blob/master/CHANGES.md#2025-09-18-2140 . You may want to already opt-in to it raising error so you can figure out if you have other such issues in your codebase.

However a VM crash is never legit, so thank you very much for the report, and I'll keep digging.

[Jiig]

Thanks for digging into this! And thanks for the link to the changelog, did not know that was coming!

[byroot]

So interestingly, what end up being corrupted isn't the hash, but the JSON_State.

diff --git a/ext/json/ext/generator/generator.c b/ext/json/ext/generator/generator.c
index 4cbe43a..9163b63 100644
--- a/ext/json/ext/generator/generator.c
+++ b/ext/json/ext/generator/generator.c
@@ -1222,7 +1222,11 @@ json_object_i(VALUE key, VALUE val, VALUE _arg)
             break;
         case T_SYMBOL:
             if (RB_UNLIKELY(arg->first_key_type != T_SYMBOL)) {
+                fprintf(stderr, "\n\nstate->space: ");
+                rb_p(state->space);
                 json_inspect_hash_with_mixed_keys(arg);
+                fprintf(stderr, "after: ");
+                rb_p(state->space);
             }
 
             key_to_s = rb_sym2str(key);

gives:

state->space: false
after: false


state->space: false
after: 3418077057644731106

So yeah, memory corruption, still need to figure how the root cause though, that won't be easy :/

[byroot]

Alright, I noticed it wasn't always failing, so I did some git bisect and it is blaming ccca602, which narrows it down quite a bit.

[byroot]

I'm a bit burnt out tonight, so to resume tomorrow (or if someone wish to dig into it), I reduced the repro to:

require 'json'
# Warning[:deprecated] = true

test_data = {
  "flag" => true,
  "data" => 10_000.times.map { [1.0] },
  :flag => false,
}

10.times do
  test_data.to_json
end

And got some debug code in: master...byroot:json:bug-929-mixed-keys

Also I'm now somehow able to repro on macOS, which should make it easier to add debug points in ruby.

[byroot]

Alright this is driving me nuts. Turns out the first two words of the JSON_Generator_state are consistently being corrupted in the repro.

state=0xa912f4230 [-1] = 0x0, [0] = 0x0, [1] = 0x0
after: tate=0xa912f4230 [-1] = 0x0, [0] = 0xcedfe39fcc487f51, [1] = 0x81c
tmp/j.rb:11: [BUG] WAHT after?

The preceding and following words are left intact. So something is writing 16B of gargage in the state struct. I have no idea how I can figure out the cause of this without using rr (which I don't have access to), except to stare at the first failing commit.

[byroot]

Alright, I figured it out even though it made no sense.

To find the offending code, I had the idea to free the state struct before invoking the routine, in the hope that the code that end up writing in it would blow up with a SEGV.

It turns out the problem was that the JSON::Generator::State (which wraps the struct) was garbage collected, and the pointer was freed a second time. So it's not so much that it was corrupted, it's that it was freed and re-used.

What makes no sense though is that JSON::Generator::State is the self argument, so it very much should be on the stack and absolutely not garbage collected.

I have a fix anyway, but I'll have to probe a few GC people because this is really wild.

[byroot]

I wanted to release the fix tonight but GitHub Actions is having an outage. That will be for tomorrow.

[Jiig]

Thank you very much for the quick fix!