-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc: add warnings about transferring Buffers and ArrayBuffer #33252
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer an approach where the user cannot get it wrong, i.e. even if they put a buffer in the transfer list, we still copy it. I'm worried that not all users would actually read this warning.
Maybe an option or flag where the user has to opt-in to this unsafe behavior, where the description in this PR applies.
I'm not all that familiar with this API. Does similar considerations apply in the web as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGTM
The only suggestion I’d have is that it would seem to make more sense to me to move this to the Buffer doc and link to that from the MessagePort
documentation, because neither the problem of multiple views over the same memory range nor the possibility an ArrayBuffer
being detached are specific to MessagePort
s or Workers.
From what I can see, #32759 only partially addresses the issue in that it only helps with @ronag ... it is not possible to completely protect users against getting it wrong here as the same fundamental problem exists with TypedArray instances in general although those are less likely to crash the process if misused. As @mafintosh points out in the other thread, the issue is the fact that |
@jasnell Not sure what you mean by “safe” here, but |
@jasnell agreed. I think I would almost go as far as stating it this way: "Unless you allocated the ArrayBuffer, safe Buffer or the TypedArray, you probably should not transfer it" |
I actually want to create similar warnings in a couple different places with different angles. In the |
@jasnell I’m okay with that approach, but in that case, I’d reduce the text here to re-scope it to what is actually relevant to transfers using |
Yes, but we also have a tendency to grab and use pointers to the raw data at the C++ level. What I mean by safe here is that buffer instances created at the C++ level may end up being used in ways that are completely invisible to the user land code. The Abort in the original post is one such example. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
It would be good to have a way to check if a buffer is pooled and if so clone manually.
@mcollina You mean other than
@jasnell Okay, that makes sense 👍 But that’s true for all |
Sorry, I meant this in the context that I've gone through a bunch of tests already with the Buffers created at the JavaScript level and know pretty much what the issues are there. I'm still going through the various ways we create buffers at the C++ level to see if there are any other possible gotchas. For instance, the following are handled differently with regards to transferability: Here, for instance, the returned static char data[10] = {};
memset(data, 1, 10);
v8::MaybeLocal<v8::Object> buf =
Buffer::New(env->isolate(), data, 10);
args.GetReturnValue().Set(buf.ToLocalChecked()); While here, it is not (the buffer will be cloned), and the original char* data = node::Malloc<char>(10);
memset(data, 0, 10);
v8::MaybeLocal<v8::Object> buf =
Buffer::New(env->isolate(), data, 10, [](char* data, void* hint) {
free(data);
}, nullptr);
args.GetReturnValue().Set(buf.ToLocalChecked()); |
This check is not really a reliable way of determining if a const ab = new ArrayBuffer(100);
const buf = Buffer.from(ab, 0, 10); // Not pooled, just over allocated.
console.log(buf.byteLength !== buf.buffer.byteLength); |
@jasnell Right, but that doesn’t seem like a typical situation to me – if the |
Or it's not and it's just a silly bug that a user does. The point is not really about how likely the case is but about how reliable of a check it is to determine if the Buffer is pooled. It fails in the following case also: const a = Buffer.from('test');
const b = Buffer.from(a.buffer);
console.log(b.byteLength === b.buffer.byteLength); In which case the byteLength's are identical but |
@jasnell Right, that second example is a good point, because I’d be more worried about false positives here than false negatives. But I’m also not sure that there would be any reasonable way to check whether |
There absolutely isn't a way which is why the recommendation here should be: never transfer a Buffer unless you know exactly where it came from and how it's being used. |
I don't know about Buffer specifics in this case, but at least on the Web upon transfer the original underlying Couldn't the same check be used here? ( In the worst case, it will match empty buffers too but these can be filtered out when Buffer is first received. |
@RReverser ... that happens here also when an |
Signed-off-by: James M Snell <[email protected]>
50da1ed
to
67a5a65
Compare
Landed in baba42c |
Signed-off-by: James M Snell <[email protected]> PR-URL: #33252 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Signed-off-by: James M Snell <[email protected]> PR-URL: #33252 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Signed-off-by: James M Snell <[email protected]> PR-URL: #33252 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
See discussion here: #33240
/cc @addaleax @mafintosh @ronag @mcollina
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes