stdio: implement manual start for ReadStream #36277

mmomtchev · 2020-11-26T11:59:40Z

All stdio ReadStream's use manual start to avoid
consuming data for example when a process
execs/spawns

Refs: #36251

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

joyeecheung · 2020-11-26T16:06:55Z

cc @nodejs/streams

ronag

manualStart should be documented.

ronag · 2020-11-26T16:31:49Z

I'm not sure I understand the root problem here. When would it ever not be safe to start reading immediately? Seems more like a workaround for a different issue.

mscdex · 2020-11-26T16:59:58Z

Minor nit: perhaps call this autoStart instead which would fit better with existing naming (e.g. autoClose) ?

mmomtchev · 2020-11-26T17:01:55Z

I'm not sure I understand the root problem here. When would it ever not be safe to start reading immediately? Seems more like a workaround for a different issue.

@ronag stdin - when reading from a file through a shell redirection, reading will start immediately and the data will be waiting to be consumed in the NodeFile buffers
This is a problem when you want to spawn a child process to read this data - look at @isaacs 's issue

@mscdex there is already a manualStart option for net.Socket

Where should this be documented? Readable or ReadStream? The net.Socket manualStart is not documented

mcollina

Could this be implemented outside of Readable? How could somebody start the stream if manualStart is true?

mcollina · 2020-11-26T22:11:19Z

lib/internal/streams/readable.js

@@ -196,7 +196,8 @@ function Readable(options) {
  Stream.call(this, options);

  destroyImpl.construct(this, () => {
-    maybeReadMore(this, this._readableState);
+    if (!options || !options.manualStart)
+      maybeReadMore(this, this._readableState);


This needs some tests that are specific to streams. How is a user of Stream be able to use this?

@mcollina @joyeecheung @ronag @mscdex

Attaching a data handler will start the stream.
I simply recreated the semantics of the non-documented option in net.Socket - which is what the thread bootstrap code uses.
Its only user at the moment is this thread bootstrap code for the stdin stream. The net.Socket option has a second use case in TLSWrap - but I am not sure if it actually works - because this wouldn't happen : #35475

The reason I did in Readable is simply that the reading is started from here for all Readable except net.Socket which has its own startup code

To move this to ReadStream I would need to reimplement part of this code and it will be a much more invasive change

So two options:

Keep both manualStart options as hidden non-documented options, eventually replacing them with Symbol names

Document them, unit test them and make them official

It is also worth noting that there are two other net.Socket specific options, both non-documented, pauseOnStart and pauseOnConnect - which have different semantics - the bootstrap code being made for manualStart

lib/internal/streams/readable.js

ronag · 2020-11-27T12:14:24Z

What about?

const myStream = createStream()
myStream.pause()

mmomtchev · 2020-11-27T12:20:14Z

@ronag For a ReadStream with a file, the pause is too late

maybeReadMore in Readable will eventually trigger a C++ FileHandle::ReadStart which will call uv_fs_read()
uv_fs_read() will happen in another thread and quite often will beat stream.pause() - or if it doesn't alright beat it - at least by the time you pause it, everything will already be set in motion and the first part of the file will be consumed

mmomtchev · 2020-11-27T12:25:28Z

What about?

const myStream = createStream()
myStream.pause()

Wait a second, just tested it and maybe you are right

mmomtchev · 2020-11-27T12:26:51Z

@ronag, yes, in fact the pause will be immediate because maybeReadMore happens in a nextTick
However if you pause, this does not cancel maybeReadMore which still schedules a read
Any ideas?

ronag · 2020-11-27T12:29:36Z

However if you pause, this does not cancel maybeReadMore which still schedules a read

Maybe we should fix this? i.e. maybeReadMore does nothing if paused.

mmomtchev · 2020-11-27T12:45:51Z

However if you pause, this does not cancel maybeReadMore which still schedules a read

Maybe we should fix this? i.e. maybeReadMore does nothing if paused.

Look at this:

node/lib/internal/streams/readable.js

Line 620 in 0a23d65

while (!state.reading && !state.ended &&

If the stream is paused, state.flowing is false - however state.length is 0 and state.highWaterMark is 65536 so the read still happens
What shall be the interaction between state.flowing and state.highWaterMark - the huge comment above that while loop does not make it very clear - but it sounds like if the stream is not flowing, try still to fill the buffer up to state.highWaterMark - which does sound somewhat weird to me? @mcollina ?

So using manualStart can be replaced by

const myStream = createStream()
myStream._readableState.highWaterMark = 0;
myStream.pause()

The above solution also works, I tested it

Is this intended? It does not feel very right

ronag · 2020-11-27T12:50:49Z

@mmomtchev maybe in the construct callback we check if paused before starting to read?

  destroyImpl.construct(this, () => {
    if (!this.isPaused()) {
      maybeReadMore(this, this._readableState);    
    }
   })

mcollina

lgtm

nodejs-github-bot · 2020-11-27T13:20:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/34585/

mmomtchev · 2020-11-27T14:12:50Z

This test
https://github.com/nodejs/node/blob/master/test/parallel/test-stream-preprocess.js
which is using 'readable' is failing because of this:

node/lib/internal/streams/readable.js

Line 873 in 0a23d65

state.flowing = false;

Does anyone know why flowing is set to false when a 'readable' is binded?

mmomtchev · 2020-11-27T14:37:42Z

I tried removing it and there is an unit test which does specifically test for this, although I suspect that it might not be intentional
Its author is referring to highWaterMark=0 streams, but in fact, this test passes because flowing is set to false when a readable is binded

https://github.com/nodejs/node/blob/master/test/parallel/test-stream-readable-hwm-0-no-flow-data.js

ronag · 2020-11-27T14:39:24Z

Does anyone know why flowing is set to false when a 'readable' is binded?

When binding 'readable' the stream should leave "flowing mode" and stop emitting 'data' events since the user is now expected to call .read().

mmomtchev · 2020-11-27T15:03:24Z

Does anyone know why flowing is set to false when a 'readable' is binded?

When binding 'readable' the stream should leave "flowing mode" and stop emitting 'data' events since the user is now expected to call .read().

'readable' stops the flowing, but does not pause the stream

Look at this test: https://github.com/nodejs/node/blob/master/test/parallel/test-stream-preprocess.js

This works because the stream continues reading - otherwise the readline interface won't work - flowing is false but the stream is not paused
After this PR, the stream will really be paused and the readline interface won't be reading

lib/internal/streams/readable.js

lib/internal/bootstrap/switches/is_main_thread.js

ronag · 2020-11-27T20:31:42Z

Maybe:

diff --git a/lib/internal/streams/readable.js b/lib/internal/streams/readable.js
index 93153908fe..4f72c0cdc1 100644
--- a/lib/internal/streams/readable.js
+++ b/lib/internal/streams/readable.js
@@ -196,7 +196,9 @@ function Readable(options) {
   Stream.call(this, options);
 
   destroyImpl.construct(this, () => {
-    maybeReadMore(this, this._readableState);
+    if (!this.isPaused()) {
+      maybeReadMore(this, this._readableState);
+    }
   });
 }
 
@@ -870,8 +872,8 @@ Readable.prototype.on = function(ev, fn) {
   } else if (ev === 'readable') {
     if (!state.endEmitted && !state.readableListening) {
       state.readableListening = state.needReadable = true;
-      state.flowing = false;
+      this.pause();
       state.emittedReadable = false;
       debug('on readable', state.length, state.reading);
       if (state.length) {
         emitReadable(this);

mmomtchev · 2020-12-01T10:00:02Z

Maybe:

diff --git a/lib/internal/streams/readable.js b/lib/internal/streams/readable.js
index 93153908fe..4f72c0cdc1 100644
--- a/lib/internal/streams/readable.js
+++ b/lib/internal/streams/readable.js
@@ -196,7 +196,9 @@ function Readable(options) {
   Stream.call(this, options);
 
   destroyImpl.construct(this, () => {
-    maybeReadMore(this, this._readableState);
+    if (!this.isPaused()) {
+      maybeReadMore(this, this._readableState);
+    }
   });
 }
 
@@ -870,8 +872,8 @@ Readable.prototype.on = function(ev, fn) {
   } else if (ev === 'readable') {
     if (!state.endEmitted && !state.readableListening) {
       state.readableListening = state.needReadable = true;
-      state.flowing = false;
+      this.pause();
       state.emittedReadable = false;
       debug('on readable', state.length, state.reading);
       if (state.length) {
         emitReadable(this);

@ronag, this is a bold move and I like it because it is a move in the right direction - having only one paused flag with uniform semantics, but I will have to modify this unit test and there will probably be user code that won't run after this change
https://github.com/nodejs/node/blob/master/test/parallel/test-stream-preprocess.js

ronag · 2020-12-02T17:04:47Z

What exactly fails with my proposal?

mmomtchev · 2020-12-02T22:01:34Z

In that unit test, once you pause it because of the readable event, nothing will resume it for the readline interface
Currently it works because the stream is not really paused, it is simply not flowing

mmomtchev · 2020-12-09T15:09:46Z

@ronag, I can't get test/parallel/test-stream-preprocess.js to work without changing it if I start messing with .pause()

mcollina · 2021-01-05T15:08:07Z

test/parallel/test-stream-manualstart.js

+    fd: fd,
+    manualStart: false
+  });
+  setTimeout(() => assert(rs.bytesRead > 0), common.platformTimeout(10));


this should be a setImmediate.

The read handler is already in a setImmediate - so this will run the assert before Node has a chance to start reading data.
I don't have any ideas on how to achieve this without a timer.

thanks for the clarification. Maybe add this in a comment?

mcollina · 2021-01-05T15:08:14Z

test/parallel/test-stream-manualstart.js

+    fd: fd,
+    manualStart: true
+  });
+  setTimeout(() => assert(rs.bytesRead === 0), common.platformTimeout(10));


his should be a setImmediate

mcollina · 2021-01-05T15:08:35Z

test/parallel/test-stream-manualstart.js

+const assert = require('assert');
+const fs = require('fs');
+
+fs.promises.open(__filename).then(common.mustCall((fd) => {


why are you using fs.promises? Could you please use fs?

Because a cook who doesn't eat its own food is not a good cook 😄

mcollina · 2021-01-05T15:09:51Z

test/parallel/test-stream-manualstart.js

+const fs = require('fs');
+
+fs.promises.open(__filename).then(common.mustCall((fd) => {
+  const rs = new fs.ReadStream(null, {


This test fs.ReadStream, not stream.Readable. While this test is correct, can you please use one that just use stream.Readable?

A Readable always starts in paused mode, this change affects only a ReadStream

A Readable always starts in paused mode

How is that relevant? It behaves the same as ReadStream. The same test should apply.

A Readable always starts in paused mode, this change affects only a ReadStream

If that was true, you'd not need to modify Readable.

Maybe there is some biggus problem, how do you think?

Maybe there is some biggus problem, how do you think?

Likely, but it's possible to add an independent test.

I'm confused what the problem here is? We just need change ReadStream to Readable. What is the "biggus" problem?

ronag

Please see #36823

mmomtchev · 2021-01-13T11:59:08Z

Included in #36823

gireeshpunathil added the stream Issues and PRs related to the stream subsystem. label Nov 26, 2020

ronag requested changes Nov 26, 2020

View reviewed changes

mcollina requested changes Nov 26, 2020

View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 27, 2020

mcollina approved these changes Nov 27, 2020

View reviewed changes

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 27, 2020

ronag reviewed Nov 27, 2020

View reviewed changes

lib/internal/streams/readable.js Outdated Show resolved Hide resolved

ronag reviewed Nov 27, 2020

View reviewed changes

lib/internal/streams/readable.js Show resolved Hide resolved

ronag reviewed Nov 27, 2020

View reviewed changes

lib/internal/bootstrap/switches/is_main_thread.js Outdated Show resolved Hide resolved

mmomtchev force-pushed the readstream-manual-start branch from 7fc79f0 to 5cd8cb6 Compare January 5, 2021 12:38

mcollina requested changes Jan 5, 2021

View reviewed changes

ronag mentioned this pull request Jan 6, 2021

stdio: lazy read ReadStream #36823

Closed

ronag reviewed Jan 6, 2021

View reviewed changes

mmomtchev closed this Jan 13, 2021

snyk-bot mentioned this pull request Sep 13, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.6.24 to 4.23.0 baby636/node#64

Open

ron-a20 mentioned this pull request Sep 13, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 ron-a20/node#73

Open

This was referenced Sep 13, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 Rednickle/node-1#53

Open

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 aliceUnhinged613/node#66

Open

wedataintelligence mentioned this pull request Sep 13, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 wedataintelligence/node#40

Open

snyk-bot mentioned this pull request Sep 13, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.2.13 to 4.23.0 kissofire/node#75

Open

molllyn1 mentioned this pull request Sep 14, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 molllyn1/node#55

Open

137717unity mentioned this pull request Sep 14, 2022

[Snyk] Security upgrade gatsby-plugin-sharp from 2.3.10 to 4.23.0 137717unity/node#64

Open

snyk-bot mentioned this pull request Sep 25, 2022

[Snyk] Fix for 22 vulnerabilities 137717unity/node#70

Open

wedataintelligence mentioned this pull request Sep 26, 2022

[Snyk] Fix for 22 vulnerabilities wedataintelligence/node#44

Open

137717unity mentioned this pull request Sep 26, 2022

[Snyk] Fix for 22 vulnerabilities 137717unity/node#73

Open

snyk-bot mentioned this pull request Sep 27, 2022

[Snyk] Fix for 21 vulnerabilities Rednickle/node-1#56

Open

wedataintelligence mentioned this pull request Sep 29, 2022

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#50

Open

This was referenced Sep 29, 2022

[Snyk] Fix for 24 vulnerabilities 137717unity/node#75

Open

[Snyk] Fix for 23 vulnerabilities Rednickle/node-1#60

Open

This was referenced Oct 5, 2022

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#51

Open

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#52

Open

snyk-bot mentioned this pull request Oct 6, 2022

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#53

Open

137717unity mentioned this pull request Oct 7, 2022

[Snyk] Fix for 24 vulnerabilities 137717unity/node#76

Open

wedataintelligence mentioned this pull request Oct 7, 2022

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#54

Open

This was referenced Oct 8, 2022

[Snyk] Fix for 24 vulnerabilities wedataintelligence/node#55

Open

[Snyk] Fix for 24 vulnerabilities 137717unity/node#77

Open

137717unity mentioned this pull request Oct 9, 2022

[Snyk] Fix for 24 vulnerabilities 137717unity/node#78

Open

snyk-bot mentioned this pull request Oct 9, 2022

[Snyk] Fix for 23 vulnerabilities Rednickle/node-1#63

Open

stdio: implement manual start for ReadStream #36277

stdio: implement manual start for ReadStream #36277

Conversation

mmomtchev commented Nov 26, 2020

Checklist

joyeecheung commented Nov 26, 2020

ronag left a comment

Choose a reason for hiding this comment

ronag commented Nov 26, 2020

mscdex commented Nov 26, 2020

mmomtchev commented Nov 26, 2020

mcollina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ronag commented Nov 27, 2020

mmomtchev commented Nov 27, 2020

mmomtchev commented Nov 27, 2020

mmomtchev commented Nov 27, 2020

ronag commented Nov 27, 2020

mmomtchev commented Nov 27, 2020

ronag commented Nov 27, 2020 • edited Loading

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Nov 27, 2020

mmomtchev commented Nov 27, 2020

mmomtchev commented Nov 27, 2020 • edited Loading

ronag commented Nov 27, 2020 • edited Loading

mmomtchev commented Nov 27, 2020

ronag commented Nov 27, 2020

mmomtchev commented Dec 1, 2020

ronag commented Dec 2, 2020 • edited Loading

mmomtchev commented Dec 2, 2020

mmomtchev commented Dec 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ronag left a comment

Choose a reason for hiding this comment

mmomtchev commented Jan 13, 2021

ronag commented Nov 27, 2020 •

edited

Loading

mmomtchev commented Nov 27, 2020 •

edited

Loading

ronag commented Nov 27, 2020 •

edited

Loading

ronag commented Dec 2, 2020 •

edited

Loading