Use @streamparser/json if the input is too large to fit in a V8 string#6016
Use @streamparser/json if the input is too large to fit in a V8 string#6016mstange wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6016 +/- ##
==========================================
- Coverage 83.82% 83.80% -0.02%
==========================================
Files 328 328
Lines 34255 34283 +28
Branches 9572 9581 +9
==========================================
+ Hits 28713 28731 +18
- Misses 5114 5124 +10
Partials 428 428 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
fatadel
left a comment
There was a problem hiding this comment.
Looks good to me personally, but Claude found a minor issue:
V8_STRING_MAX_SIZE is slightly above the actual V8 limit. The constant is 512 * 1024 * 1024 - 1 = (1 << 29) - 1 = 536870911. V8's String::kMaxLength under pointer compression (the typical config — buffer.constants.MAX_STRING_LENGTH exposes this in Node) is (1 << 29) - 24 = 536870888. That leaves a ~23-byte window where the size check passes but the ASCII-decoded string would still throw RangeError: Invalid string length. Practically irrelevant, but the fix is trivial: (1 << 29) - 24, or just import buffer.constants.MAX_STRING_LENGTH in Node paths.
|
Hah, oops. |
Here's a file which decompresses to a 605.8MB JSON string: etw-speedometer3.json.gz
Loading this file in Chrome, or in node via profiler-edit, fails due to the string size.
As a workaround we can use a streaming JSON parser which doesn't materialize a string for the entire thing.