Document (or fix?) v8.deserialize' 2gb limitation for input buffer
#40059
Comments
|
After tracking for a while, I found that the constructor of See line Lines 3271 to 3283 in 5c1adda I've tried to simply use ValueDeserializer::ValueDeserializer(Isolate* isolate, const uint8_t* data,
size_t size, Delegate* delegate) {
- if (base::IsValueInRangeForNumericType<int>(size)) {
+ if (base::IsValueInRangeForNumericType<size_t>(size)) {
private_ = new PrivateData(
reinterpret_cast<i::Isolate*>(isolate),
- base::Vector<const uint8_t>(data, static_cast<int>(size)), delegate);
+ base::Vector<const uint8_t>(data, static_cast<size_t>(size)), delegate);
} else {
private_ =
new PrivateData(reinterpret_cast<i::Isolate*>(isolate),
base::Vector<const uint8_t>(nullptr, 0), nullptr);
private_->has_aborted = true;
}
} but get Let me explain why we get this error. When we try to deserialize a huge buffer, we need to construct a node/deps/v8/src/objects/value-serializer.cc Lines 1114 to 1122 in 55379eb HOWEVER, node/deps/v8/src/objects/value-serializer.cc Lines 1142 to 1153 in 55379eb Perhaps we can just leave everything original, if we are okay to be unable to deserialize huge objects successfully serialized by v8. This is just a little bit weird, but it is safe. Or, we can make every object serialized by Possible Solution:
You can pull rayw000@d556d61 and #40243 for test. This following code snippet #!/usr/bin/env node --max-old-space-size=32768
const v8 = require('v8');
const str = `AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA`
const obj = {}
for (let i = 1; i < 22; i++) {
for (let j = 0; j < 2 ** i; j++) {
obj[j] = str;
}
}
const buffer = v8.serialize(obj)will output |
|
I'm kinda +1 with @rayw000's idea, we could just remove the limitation and allow the deserialization of all the objects that can be serialized and document it somewhere. @joyeecheung @addaleax what do you both think? |
|
I just submitted a patch to Gerrit: https://chromium-review.googlesource.com/c/v8/v8/+/3170411 |
1. Now there is no serializer/deserializer-specific buffer size limit. 2. Update AUTHORS Ref: nodejs/node#40059 Change-Id: Iad4c6d8f68a91ef21d3c404fb7945949e69ad9e2 Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3170411 Reviewed-by: Jakob Kummerow <[email protected]> Reviewed-by: Clemens Backes <[email protected]> Commit-Queue: Jakob Kummerow <[email protected]> Cr-Commit-Position: refs/heads/main@{#77084}
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
Original commit message:
[deserialization] Remove unnecessarily limit on buffer size
1. Now there is no serializer/deserializer-specific buffer size limit.
2. Update AUTHORS
Ref: #40059
Change-Id: Iad4c6d8f68a91ef21d3c404fb7945949e69ad9e2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3170411
Reviewed-by: Jakob Kummerow <[email protected]>
Reviewed-by: Clemens Backes <[email protected]>
Commit-Queue: Jakob Kummerow <[email protected]>
Cr-Commit-Position: refs/heads/main@{#77084}
Refs: v8/v8@422dc37
PR-URL: #40450
Fixes: #40059
Reviewed-By: Michaël Zasso <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
|
Since #40450 and #40243 are landed, now we can serialize any objects requiring buffer no larger than |
Original commit message:
[deserialization] Remove unnecessarily limit on buffer size
1. Now there is no serializer/deserializer-specific buffer size limit.
2. Update AUTHORS
Ref: #40059
Change-Id: Iad4c6d8f68a91ef21d3c404fb7945949e69ad9e2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3170411
Reviewed-by: Jakob Kummerow <[email protected]>
Reviewed-by: Clemens Backes <[email protected]>
Commit-Queue: Jakob Kummerow <[email protected]>
Cr-Commit-Position: refs/heads/main@{#77084}
Refs: v8/v8@422dc37
PR-URL: #40450
Fixes: #40059
Reviewed-By: Michaël Zasso <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
PR-URL: #40243 Fixes: #40059 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michaël Zasso <[email protected]>
Version
v16.9.0
Platform
Darwin theodore 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64
Subsystem
v8
What steps will reproduce the bug?
Here is a script that will demonstrate the issue:
And here is the output on my system:
How often does it reproduce? Is there a required condition?
100% of the time if the buffer to deserialize exceeds 2gb.
What is the expected behavior?
Either the documentation is updated to reflect the limit or the limit is removed.
What do you see instead?
Please see the output above.
Additional information
v8's
serializeanddeserializeare fantastic and very performant for large datasets; much faster than the alternatives like e.g.msgpackr. I'd love to continue using them as my dataset grows so that I can put off re-architecting things a bit further, but I realize it's probably a very extreme use case. Documenting the limitation will save the next person who hits it some time, however! I spent a lot of time assuming that my file writing code was broken in some way (which I had to rewrite because it also has a 2gb limitation; I hit both limits at the same time but didn't realizedeserializehad the same limit).In the short term I will probably shard keys across multiple files.
The text was updated successfully, but these errors were encountered: