-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream: 'readable' have precedence over flowing #18994
Conversation
c2fb53b
to
53ef8f2
Compare
Wouldn't |
@mscdex you are right. I didn't account for https://github.com/nodejs/node/blob/master/lib/events.js#L324-L336. Do you think I can just listen for |
@mcollina so basically just suspends flowing when readable is set? |
@mafintosh that's the goal yes. It also restores the behavior after |
@mcollina wondering if we should even fix this. the same behaivor described in the issue would happen if there are two consumers of the readable event anyway (which is what happens in flowing mode). not too much of a fan of implicit event listener side effects |
I am good with not fixing it. But it’s a discussion to have. This fixed a major usability problem if you want to have both a on(‘data’) and on(‘readable’)/read() at the same time (with pipe) read() will always return null. I’m ok if we want to remove the resume side effect when removing ‘readable’, as that might be confusing. However we should really be updating readableListening: this is also a bug without the fix for read() itself. |
Removing the removeListener stuff but keeping the other part sounds like a good middleway to me 👍 |
@nodejs/streams @mafintosh PTAL |
@nodejs/tsc what do you think? An alternative approach might be to remove the |
doc/api/stream.md
Outdated
@@ -801,6 +811,10 @@ In general, the `readable.pipe()` and `'data'` event mechanisms are easier to | |||
understand than the `'readable'` event. However, handling `'readable'` might | |||
result in increased throughput. | |||
|
|||
If both `'readable'` and [`'data'`][] are used at the same time, `'readable'` | |||
takes precedence in controlling the flow, i.e. `'data'` will be emitted | |||
only when [`stream.read()`][stream-read] is called |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A nit: missing period.
@@ -35,6 +35,7 @@ let expectEndingData = expectTotalData; | |||
const r = new Readable({ highWaterMark: 1000 }); | |||
let chunks = totalChunks; | |||
r._read = function(n) { | |||
console.log('_read called', chunks); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
did you mean to add these logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, they follow the prevailing pattern in this test file. Without the console.log
statements, this is undebuggable.
Lots of good tests! LGTM from me 👍 |
03e66ed
to
44fad80
Compare
Tagging @nodejs/tsc because it is semver-major. |
@@ -770,10 +774,16 @@ cause some amount of data to be read into an internal buffer. | |||
|
|||
```javascript | |||
const readable = getReadableStreamSomehow(); | |||
readable.on('readable', () => { | |||
readable.on('readable', function() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: not sure why this was changed and it doesn't really matter but for consistency I would keep the arrow function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lpinca it is using this.read() below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, ignore me.
44fad80
to
8085e2e
Compare
doc/api/stream.md
Outdated
- version: REPLACEME | ||
pr-url: FILLMEIN | ||
description: > | ||
using 'readable' requires calling .read(). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you avoid the line break and capitalize the sentence?
doc/api/stream.md
Outdated
- version: REPLACEME | ||
pr-url: FILLMEIN | ||
description: > | ||
resume has no effect if there is a 'readable' event listening |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
lib/_stream_readable.js
Outdated
@@ -223,6 +223,7 @@ Readable.prototype.unshift = function(chunk) { | |||
}; | |||
|
|||
function readableAddChunk(stream, chunk, encoding, addToFront, skipChunkCheck) { | |||
debug('readableAddChunk'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be good to have chunk
printed out here as well
lib/_stream_readable.js
Outdated
|
||
if (ev === 'data') { | ||
// update readableListening | ||
state.readableListening = this.listenerCount('readable') > 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you change to comment above this line so that it says why we do this, rather than what we do? I assume it’s about the this.resume()
call below?
In Streams3 the 'readable' event/.read() method had a lower precedence than the `'data'` event that made them impossible to use them together. This make `.resume()` a no-op if there is a listener for the `'readable'` event, making the stream non-flowing if there is a `'data'` listener. Fixes: nodejs#18058
8085e2e
to
578a7b9
Compare
@addaleax PTAL |
Landed as cf5f986 |
In Streams3 the 'readable' event/.read() method had a lower precedence than the `'data'` event that made them impossible to use them together. This make `.resume()` a no-op if there is a listener for the `'readable'` event, making the stream non-flowing if there is a `'data'` listener. Fixes: #18058 PR-URL: #18994 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
Seems like failures in cf5f986#diff-573b8412079b987e160a3fde511d5b9b are showing up in a couple places In a couple CI runs in #19201 and also in http://github.com/nodejs/node/pull/19924 @mcollina thoughts?
|
@MylesBorins We are already tracking it in #19905. |
Caused by nodejs/node#18994 Closes #26
In Streams3 the 'readable' event/.read() method had a lower precedence
than the
'data'
event that made them impossible to use them together.This make
.resume()
a no-op if there is a listener for the'readable'
event, making the stream non-flowing if there is a'data'
listener.Fixes: #18058
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesAffected core subsystem(s)
stream, http