streams: support unlimited synchronous cork/uncork cycles #6164
Conversation
mcollina
added
stream
|
@jfhbrook are you able to test this? |
|
cc @nodejs/streams please review asap. @thealphanerd you might want to include this one for #6149. |
| @@ -387,12 +387,17 @@ function clearBuffer(stream, state) { | |||
|
|
|||
| doWrite(stream, state, true, state.length, buffer, '', holder.finish); | |||
|
|
|||
| // doWrite is always async, defer these to save a bit of time | |||
| // doWrite is almost always async, defer these to save a bit of time | |||
| // as the hot path ends with doWrite | |||
| state.pendingcb++; | |||
| state.lastBufferedRequest = null; | |||
| state.corkedRequestsFree = holder.next; | |||
There was a problem hiding this comment.
probably want to remove this line as well
|
Other than @calvinmetcalf's comment this is LGTM from em |
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: nodejs@89aeab9 (nodejs#4354). Fixes: nodejs#6154.
mcollina
force-pushed
the
unlimited-corked-requests
branch
from
f91d9b5 to
ce2a040
Compare
|
I have updated it, mainly cleaning things up. Let me know what you think cc @mafintosh @calvinmetcalf. |
|
(how do I launch CITGM?) |
|
@mcollina The |
|
Another CI: https://ci.nodejs.org/job/node-test-pull-request/2247 (previous one missing the "safe" flag). |
|
One failure (seems spurious) on ARM https://ci.nodejs.org/job/node-test-binary-arm/1691/RUN_SUBSET=6,nodes=pi2-raspbian-wheezy/tapTestReport/ |
A little hard because we're only seeing this in production, but I'll see what we can do. |
|
CIGTM passes. |
|
How long will it take for this to land in v5? |
|
Done for me, awaiting LGTMs. CI and CIGTM passes. Today or tomorrow I presume. |
| res = conn.write(buf); | ||
| conn.uncork(); | ||
| } | ||
| assert.equal(i, N); |
There was a problem hiding this comment.
maybe also assert res is still true, since it could have been false in the last iteration.
There was a problem hiding this comment.
You are right, it's not really an assert on that. I was also afraid of putting the assert in i, as it could be spurious (it depends on OS buffers and the like). I would avoid to stretching luck, I picked 100 as a random number that was working on my box.
There was a problem hiding this comment.
100 seems reasonable, a comment there would be nice but I guess it's pretty straightforward as is.
|
LGTM, nice fix. |
More importantly, v4? |
|
@jfhbrook typically pull requests wait for 2 days before being landed so all collaborators have a chance to look at them. After that the road to v4 and v5 is usually very fast. The fix is good and it has already been LGTMd by 2 collaborators so it shouldn't be more than 2 days unless new information is discovered. Of course - in cases of urgency pull requests might be merged sooner, and also - you can always take the changes here - apply them to the v4 branch and build it if you need this fix faster than that. |
|
If there aren't any other concern, I would merge it on Thursday morning (CEST). |
|
@mcollina LGTM |
|
@mcollina also great job on the fast turn around! |
|
LGTM |
|
@mcollina just a heads up, we are in fact floating this patch in prod right now, so far no breakage \o/ This is expected to land in 4.4.4 right? Just trying to track the work around not having to float patches anymore :) |
|
@jfhbrook if you're eager and want to help you're welcome to open a pull request backporting this patch against the v4-staging branch. Contributions are always welcome. |
|
Sure. Would that be helpful? Like in our case I just did a cherry-pick |
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: 89aeab9 (#4354). Fixes: #6154 PR-URL: #6164 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: James M Snell <[email protected]>
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: 89aeab9 (#4354). Fixes: #6154 PR-URL: #6164 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: James M Snell <[email protected]>
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: 89aeab9 (#4354). Fixes: #6154 PR-URL: #6164 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: James M Snell <[email protected]>
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) #5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) #6279 * update ESLint to 2.7.0 (silverwind) #6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) #6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) #6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) #6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behaviour was in node previously and was not removed intentionally. (Rich Trott) #6090 src: * add SIGINFO to supported signals (James Reggio) #6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) #6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) #6069
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) #5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) #6279 * update ESLint to 2.7.0 (silverwind) #6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) #6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) #6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) #6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behavior was in node previously and was not removed intentionally. (Rich Trott) #6090 src: * add SIGINFO to supported signals (James Reggio) #6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) #6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) #6069
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) #5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) #6279 * update ESLint to 2.7.0 (silverwind) #6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) #6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) #6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) #6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behaviour was in node previously and was not removed intentionally. (Rich Trott) #6090 src: * add SIGINFO to supported signals (James Reggio) #6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) #6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) #6069 PR-URL: #6322
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) #5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) #6279 * update ESLint to 2.7.0 (silverwind) #6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) #6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) #6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) #6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behaviour was in node previously and was not removed intentionally. (Rich Trott) #6090 src: * add SIGINFO to supported signals (James Reggio) #6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) #6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) #6069 PR-URL: #6322
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) nodejs#5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) nodejs#6279 * update ESLint to 2.7.0 (silverwind) nodejs#6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) nodejs#6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) nodejs#6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) nodejs#6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behaviour was in node previously and was not removed intentionally. (Rich Trott) nodejs#6090 src: * add SIGINFO to supported signals (James Reggio) nodejs#6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) nodejs#6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) nodejs#6069 PR-URL: nodejs#6322
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: 89aeab9 (#4354). Fixes: #6154 PR-URL: #6164 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: James M Snell <[email protected]>
Buffer: * Buffer.prototype.compare can now compare sub-ranges of two Buffers (James M Snell) #5880 deps: * update to http-parser 2.7.0 (Fedor Indutny) #6279 * update ESLint to 2.7.0 (silverwind) #6132 net: * adds support for passing DNS lookup hints to createConnection() (Colin Ihrig) #6000 node: * Make the builtin libraries available for the --eval and --print CLI options (Anna Henningsen) #6207 npm: * upgrade npm to 3.8.6 (Kat Marchán) #6153 repl: * Pressing enter in the repl will repeat the last command by default if no input has been received. This behaviour was in node previously and was not removed intentionally. (Rich Trott) #6090 src: * add SIGINFO to supported signals (James Reggio) #6093 streams: * Fix a regression that caused by net streams requesting multiple chunks synchronously when combined with cork/uncork (Matteo Collina) #6164 zlib: * The flushing flag is now configurable allowing for decompression of partial data (Anna Henningsen) #6069 PR-URL: #6322
net streams can request multiple chunks to be written in a synchronous fashion. If this is combined with cork/uncork, en error is currently thrown because of a regression introduced in: 89aeab9 (#4354). Fixes: #6154 PR-URL: #6164 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Mathias Buus <[email protected]> Reviewed-By: James M Snell <[email protected]>
Checklist
Affected core subsystem(s)
streams, writable
Description of change
net streams can request multiple chunks to be written in a synchronous
fashion. If this is combined with cork/uncork, en error is currently
thrown because of a regression introduced in:
89aeab9
(#4354).
I am not creating a full linked list anyway, because that is not needed in any "normal" operation of the stream, and I think multiple cork/uncork synchronously it is a very edge case: we can possibly waste some objects and callbacks.
Just to clarify, we are supporting a "feature" that can easily lead to event loop starvation and memory leaks, because for each synchronous cork/uncork cycle, we are adding a task in the
nextTickqueue:node/lib/_stream_writable.js
Line 344 in 819b2d3
Fixes: #6154.