stream: ensure Stream.pipeline re-emits errors without callback #20437

phated · 2018-04-30T22:15:25Z

Fixes an issue where Stream.pipeline wouldn't re-emit errors
on a stream if no callback was specified, thus swallowing
said errors.

Fixes: #20303

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

phated · 2018-04-30T22:16:24Z

cc @addaleax @mafintosh

addaleax · 2018-04-30T22:22:55Z

lib/internal/streams/pipeline.js

+    // TODO: Should eos cleanup after itself?
+    // Ref https://github.com/mafintosh/end-of-stream/pull/12
+    cleanup();
+


This seems to make sense, but I find it hard to tell how this part of the change relates to the rest. It feels like this should be something that comes with tests too?

@addaleax so i stumbled around for ~2 days on this. Without cleaning up after end-of-stream, my re-emitted error was still being handled by the error handler that end-of-stream attached and I couldn't get an uncaughtException to be emitted. Once I added the cleanup, it emitted with no issues.

Would that go in this PR? A separate commit? I'm not sure the process here.

Okay, makes sense. I think single PR and single commit is fine (but especially in the case of splitting into commits that correspond to logical changes you can use your own judgement). I think you can feel free to add a comment here that describes how this relates to the error handling, but this should be fine as-is.

I've forced pushed an updated comment that describes what I've explained above.

addaleax · 2018-04-30T22:39:11Z

CI: https://ci.nodejs.org/job/node-test-commit/18151/

mafintosh · 2018-05-04T07:59:00Z

Personally I'm 👎 on this. Emitting errors on other streams can have unintended side effects, which is why pipeline/pump tries not to do that in the first place. If all streams were "modern" you could call destroy(err) on them but thats not gonna behave well in streams < node 10 unless they have a custom destroy handler. Changing behavior based whether a cb is passed or not can lead to tricky bugs imo.

If people fell strongly about this I'd make the noop cb throw the error instead.

addaleax · 2018-05-04T08:04:26Z

@mafintosh I with @phated in that I think we should definitely do something that does not just swallow the errors silently…

jasnell · 2018-05-04T08:25:46Z

I agree with @mafintosh that this is likely not the best approach. Perhaps if pipeline() is not given a callback, a default non-op callback that throws when err is not null would be a better approach?

mafintosh · 2018-05-04T08:37:52Z

@addaleax I wouldn't say errors are swallowed silently. On error the entire pipeline is shut down and destroy handlers are called on all streams. If you wanna be notified about this outside of your stream, you can pass a cb. Lots of use-cases of pump I've seen in the wild don't pass a cb, so if pipeline were to help them they'd have to pass in noop's.

I don't feel strongly though about that part (I do about the re-emitting errors part tho), as pump still exists and gonna keep that old behaivor.

phated · 2018-05-04T18:42:28Z

I'm fine if we avoid re-emitting the error and switch it to throwing in the noop - all I really care about is that the error is actually surfaced when there's no way to watch for it (not passing a callback).

Should I update the PR?

BridgeAR · 2018-05-05T13:43:18Z

@jasnell if I understand your approach correct you want to do:

pipeline()

function pipeline(callback) {
  callback = callback || (err) => {
    if (err)
      throw err;
  }
  // Do stuff
}

To me this does not seem like a good approach since this will always end up as uncaught exception. Or did I misunderstand something?

I personally suggest to just force the user to always add a callback. If non is added, we just throw an error as we do in other APIs as well. If the user wants to ignore errors, that is totally fine.

phated · 2018-05-13T21:14:19Z

@jasnell @addaleax should I update the PR to rethrow in the noop callback?

addaleax · 2018-05-13T21:17:22Z

@phated I would do that, yes. 👍 It seems like that’s at least something that you, me, @jasnell and @mafintosh can be behind.

I personally suggest to just force the user to always add a callback.

@BridgeAR Also okay with me, but maybe for a follow-up PR since that’s tightening up behaviour even more?

addaleax

Still LGTM

CI: https://ci.nodejs.org/job/node-test-commit/18457/

BridgeAR · 2018-05-18T11:10:20Z

@addaleax

but maybe for a follow-up PR since that’s tightening up behaviour even more

It does tighten the current rule but I highly prefer that over the current PR implementation. I think throwing async is a really bad idea. Using emit would be a way the user could deal with it properly but always triggering uncaught exceptions is 😕

addaleax · 2018-05-18T11:15:14Z

Using emit would be a way the user could deal with it properly but always triggering uncaught exceptions is 😕

I agree that generating uncaught exceptions is 😕, but it’s something for which users have a clear way to address.

More practically speaking, I think this PR could pass as not being semver-major (because there are cases where it’s known that no error is going to occur, and for the other cases this is effectively not silencing errors that should not be silenced, so a bug fix) whereas making the callback non-optional is definitely semver-major imo.

jasnell · 2018-05-23T18:23:32Z

Where are we at on this one? Definitely needs a rebase.

phated · 2018-05-23T18:28:27Z

This has been sitting for approval and feedback for a long time and I probably won't have time to rebase this coming up. Can someone else take that on?

mcollina · 2018-05-23T22:09:55Z

@mafintosh are you still -1 on this?

mafintosh · 2018-05-24T09:01:32Z

@mcollina I think there are plenty of valid usecases where you don't want a cb because all error handling is contained in the pipeline (like a simple pipeline(fs.createReadStream(), httpRes)) so 👎 on that still.

I wouldn't block a merge though if people felt like it's super important.

This change https://github.com/nodejs/node/pull/20437/files#diff-eefad5e8051f1634964a3847b691ff84R49 should not be there IMO though as it makes error handling in practice tricky. With that change if a stream emits an error after end you'll have to setup your own error handling which in practice means you need to do that all the time rendering the API not that useful in many scenarios.

mcollina

Marking my request of change explicit, so it does not get landed accidentally.

mcollina · 2018-05-24T09:06:23Z

lib/internal/streams/pipeline.js

+    //
+    // TODO: Should eos cleanup after itself?
+    // Ref https://github.com/mafintosh/end-of-stream/pull/12
+    cleanup();


I think this should not be done here. 'error' can still happen after 'end', making error handling more tricky.

BridgeAR · 2018-05-29T20:16:10Z

@mafintosh as a general thing: you would be strongly against requiring a callback all the time? This is the default for pretty much everything async in Node.js and it would also solve this issue. Every user could easily just add a no-op callback in case they want to ignore that. Adding an explicit documentation about this should probably sufficient to explain this to users, don't you think so?

mafintosh · 2018-05-29T20:30:17Z

@BridgeAR not strongly against it, just personal preference.

Fixes an issue where Stream.pipeline wouldn't re-throw errors on a stream if no callback was specified, thus swallowing said errors. Fixes: nodejs#20303

phated · 2018-05-30T20:01:50Z

@addaleax @mcollina I've made rebased and made the desired changes. Can we get this wrapped up so I don't need to spend even more time rebasing and stuff?

mcollina

LGTM

mcollina · 2018-05-30T20:17:20Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15171/

mcollina · 2018-05-30T20:18:16Z

This is ready to land with a green CI.

mafintosh · 2018-05-30T21:22:43Z

👍

addaleax · 2018-05-31T09:26:59Z

Landed in c1012b4, thanks for the PR!

Fixes an issue where Stream.pipeline wouldn't re-throw errors on a stream if no callback was specified, thus swallowing said errors. Fixes: #20303 PR-URL: #20437 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

Notable Changes: * **deps**: - update V8 to 6.7.288.43 (Michaël Zasso) #19989 * **stream**: - ensure Stream.pipeline re-throws errors without callback (Blaine Bublitz) #20437 PR-URL: #21167

phated mentioned this pull request Apr 30, 2018

Cleanup before callback mafintosh/end-of-stream#12

Open

addaleax approved these changes Apr 30, 2018

View reviewed changes

phated force-pushed the stream-pipeline-reemit-errors branch from a490a89 to d42c4d4 Compare April 30, 2018 22:36

addaleax added the stream Issues and PRs related to the stream subsystem. label Apr 30, 2018

phated force-pushed the stream-pipeline-reemit-errors branch from d42c4d4 to c636365 Compare May 13, 2018 21:21

addaleax approved these changes May 13, 2018

View reviewed changes

mcollina requested changes May 24, 2018

View reviewed changes

stream: ensure Stream.pipeline re-throws errors without callback

44d3fe3

Fixes an issue where Stream.pipeline wouldn't re-throw errors on a stream if no callback was specified, thus swallowing said errors. Fixes: nodejs#20303

phated force-pushed the stream-pipeline-reemit-errors branch from c636365 to 44d3fe3 Compare May 30, 2018 20:00

mcollina approved these changes May 30, 2018

View reviewed changes

mcollina added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label May 30, 2018

mafintosh approved these changes May 30, 2018

View reviewed changes

addaleax closed this May 31, 2018

MylesBorins mentioned this pull request Jun 6, 2018

v10.4.0 proposal #21167

Merged

mcollina mentioned this pull request May 27, 2020

Promise-friendly stream.pipeline and stream.finished #33582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: ensure Stream.pipeline re-emits errors without callback #20437

stream: ensure Stream.pipeline re-emits errors without callback #20437

phated commented Apr 30, 2018

phated commented Apr 30, 2018

addaleax Apr 30, 2018

phated Apr 30, 2018

addaleax Apr 30, 2018

phated Apr 30, 2018

addaleax commented Apr 30, 2018

mafintosh commented May 4, 2018

addaleax commented May 4, 2018

jasnell commented May 4, 2018

mafintosh commented May 4, 2018

phated commented May 4, 2018

BridgeAR commented May 5, 2018 •

edited

Loading

phated commented May 13, 2018

addaleax commented May 13, 2018

addaleax left a comment

BridgeAR commented May 18, 2018

addaleax commented May 18, 2018

jasnell commented May 23, 2018

phated commented May 23, 2018

mcollina commented May 23, 2018

mafintosh commented May 24, 2018

mcollina left a comment

mcollina May 24, 2018

BridgeAR commented May 29, 2018

mafintosh commented May 29, 2018

phated commented May 30, 2018

mcollina left a comment

mcollina commented May 30, 2018

mcollina commented May 30, 2018

mafintosh commented May 30, 2018

addaleax commented May 31, 2018

stream: ensure Stream.pipeline re-emits errors without callback #20437

stream: ensure Stream.pipeline re-emits errors without callback #20437

Conversation

phated commented Apr 30, 2018

Checklist

phated commented Apr 30, 2018

addaleax Apr 30, 2018

Choose a reason for hiding this comment

phated Apr 30, 2018

Choose a reason for hiding this comment

addaleax Apr 30, 2018

Choose a reason for hiding this comment

phated Apr 30, 2018

Choose a reason for hiding this comment

addaleax commented Apr 30, 2018

mafintosh commented May 4, 2018

addaleax commented May 4, 2018

jasnell commented May 4, 2018

mafintosh commented May 4, 2018

phated commented May 4, 2018

BridgeAR commented May 5, 2018 • edited Loading

phated commented May 13, 2018

addaleax commented May 13, 2018

addaleax left a comment

Choose a reason for hiding this comment

BridgeAR commented May 18, 2018

addaleax commented May 18, 2018

jasnell commented May 23, 2018

phated commented May 23, 2018

mcollina commented May 23, 2018

mafintosh commented May 24, 2018

mcollina left a comment

Choose a reason for hiding this comment

mcollina May 24, 2018

Choose a reason for hiding this comment

BridgeAR commented May 29, 2018

mafintosh commented May 29, 2018

phated commented May 30, 2018

mcollina left a comment

Choose a reason for hiding this comment

mcollina commented May 30, 2018

mcollina commented May 30, 2018

mafintosh commented May 30, 2018

addaleax commented May 31, 2018

BridgeAR commented May 5, 2018 •

edited

Loading