Cleanup `_writableState` and `_readableState` access across codebase #445

vkurchatkin · 2015-01-15T16:18:56Z

This is a meta-issue to keep to track of all usages of _writableState and _readableState outside of streams source. These properties are considered private and should not be used unless absolutely necessary. Usage of them can indicate a few things:

the code can be rewritten using existing documented API to achieve the same result;
streams lack some consumer functionality and new public API should be introduced;
streams lack some implementor functionality and new protected API should be introduced;
documentation needs to be added for some parts of private state for implementors;
it is an optimization that is and always be possible only in core.

The list of all _writableState and _readableState usages:

`src/node.js`

L564 stdin._readableState.reading = false. Added in commit: bb56dcc by @isaacs; (src: nix stdin _readableState.reading manipulation #454)
L573 stdin._readableState.reading = false. Added in commit: bb56dcc by @isaacs; (src: nix stdin _readableState.reading manipulation #454)

`lib/_debug_agent.js`

L87 this._readableState.objectMode = true (_debug_agent: use readableObjectMode option for client stream #270);

`lib/_http_server.js`

L348 socket._readableState.flowing = null (TODO(isaacs): Need a way to reset a stream to fresh state IE, not flowing, and not explicitly paused.). Added in 967b5db by @isaacs;
L408 var needPause = socket._writableState.needDrain. Added in 085dd30 by @isaacs ;
L445 req._readableState.resumeScheduled. Not sure where this originated, but in 2efe4ab @indutny added oldMode check here;

`lib/_tls_legacy.js`

L421 this._writableState.finished;
L508 self._readableState.length > 0;

`lib/_tls_wrap.js`

L311 self._writableState.errorEmitted (Cleanup stream state in net #465);
L313 self._writableState.errorEmitted = true (Cleanup stream state in net #465);
L350 socket._readableState.length;

`lib/child_process.js`

L1061 stream._readableState.flowing (child_process: remove redundant condition #511);

`lib/crypto.js`

whole LazyTransform thing. Is it really necessary? Maybe it should go to stream? Maybe it should be public? Maybe transforms should be lazy by default?;
L56 this._writableState.decodeStrings = false;
L57 this._writableState.defaultEncoding = 'binary';
L90 var encoding = this._readableState.encoding || 'buffer' (crypto: remove use of this._readableState #610);

`lib/fs.js`

L1624 allocNewPool(this._readableState.highWaterMark);

`lib/net.js`

`lib/zlib.js`

List of used properties:

`_readableState`

reading;
objectMode;
flowing (boolean?) is used to determine which mode readable stream is in; can be true, false or null; `null is the initial state which means that is implicitly paused;
resumeScheduled;
length;
encoding;
highWaterMark;
ended;
endEmitted;

`_writableState`

needDrain;
ended;
ending;
finished;
errorEmitted;
decodeStrings;
defaultEncoding;
length;
getBuffer();

/cc @chrisdickinson

The text was updated successfully, but these errors were encountered:

chrisdickinson · 2015-01-15T16:59:37Z

This is great! Thanks for compiling a comprehensive list. The most egregious overreaches, I think, are any of the places we reach in and directly modify state. Informational access is less bad.

I'd like to encourage folks to pick these off one at a time. For changes that might require a new internal API, it'd be great to briefly discuss plans here first before cutting a PR.

On Jan 15, 2015, at 8:18 AM, Vladimir Kurchatkin [email protected] wrote:

This is a meta-issue to keep to track of all usages of _writableState and _readableState outside of streams source. These properties are considered private and should not be used unless absolutely necessary. Usage of them can indicate a few things:

the code can be rewritten using existing documented API to achieve the same result;
streams lack some consumer functionality and new public API should be introduced;
streams lack some implementor functionality and new protected API should be introduced;
documentation needs to be added for some parts of private state for implementors;
it is an optimization that is and always be possible only in core.
The list of all _writableState and _readableState usages:

src/node.js

L564 stdin._readableState.reading = false;
L573 stdin._readableState.reading = false;
lib/_debug_agent.js

L87 this._readableState.objectMode = true (#270);
lib/_http_server.js

L348 socket._readableState.flowing = null (TODO(isaacs): Need a way to reset a stream to fresh state IE, not flowing, and not explicitly paused.);
L408 var needPause = socket._writableState.needDrain;
L445 req._readableState.resumeScheduled;
lib/_tls_legacy.js

L421 his._writableState.finished;
L508 self._readableState.length > 0;
lib/_tls_wrap.js

L311 self._writableState.errorEmitted;
L313 self._writableState.errorEmitted = true;
L350 socket._readableState.length;
lib/child_process.js

L1061 stream._readableState.flowing;
lib/crypto.js

whole LazyTransform thing. Is it really necessary? Maybe it should go to stream? Maybe it should be public? Maybe transforms should be lazy by default?;
L56 this._writableState.decodeStrings = false;
L57 this._writableState.defaultEncoding = 'binary';
L90 var encoding = this._readableState.encoding || 'buffer';
lib/fs.js

L1624 allocNewPool(this._readableState.highWaterMark);
lib/net.js

L162 this._writableState.decodeStrings = false;
L174 this._readableState.flowing = false;
L196 this._readableState.ended;
L226 self._readableState.ended;
L242 this._readableState.ended = true (comment: ended should already be true, since this is called after the EOF errno and onread has eof'ed);
L243 this._readableState.endEmitted;
L362 this._writableState.length;
L392 this._readableState.endEmitted;
L405 socket._writableState.length;
L415 if (this._writableState.finished);
L429 self._writableState.errorEmitted;
L433 self._writableState.errorEmitted = true;
L535 self._readableState.length === 0;
L715 state.getBuffer();
L842 this._readableState.reading = false;
L843 this._readableState.ended = false;
L844 this._readableState.endEmitted = false;
L845 this._writableState.ended = false;
L846 this._writableState.ending = false;
L847 this._writableState.finished = false;
L848 this._writableState.errorEmitted = false;
/cc @chrisdickinson

—
Reply to this email directly or view it on GitHub.

Use public `readableObjectMode` option to construct `Transform` instead of accessing private `_readableState.objectMode`. Partially addresses #445. PR-URL: #270 Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Bert Belder <[email protected]> Reviewed-By: Chris Dickinson <[email protected]>

chrisdickinson · 2015-01-15T19:45:18Z

Of note: I'm not opposed to all non-streams access to _readableState, just accesses that are directly inspecting a property or manipulating state. APIs private to io.js itself are okay to hang off of ReadableState, but I'd like to make sure that there's a clear API there, and not just random attribute access/manipulation.

vkurchatkin · 2015-01-15T20:04:37Z

We should also probably inspect userland modules to find out what people are doing with internals.

chrisdickinson · 2015-01-15T20:42:05Z

As a heads up, I edited your issue comment to link #454 for the node.js _readableState use (sorry sorry). In the future, I can ping you instead if you'd like to keep the list maintained!

vkurchatkin · 2015-01-15T20:49:49Z

@chrisdickinson fine by me. But think it's a good idea to reference this issue so we can see the timeline of all relevant stuff here

chrisdickinson · 2015-01-15T20:53:01Z

We might add lib/zlib.js:

L423: ws.ended
L426: ws.ending
L429: ws.needDrain
L460: ws.ending || ws.ended
L461: ws.length
L479: ws.length

Per nodejs#445 this removes a reference to this._readableState in hash._flush. It was used to get the encoding on the readable side to pass to the writable side but omiting it just causes the stream to handle the encoding issues.

Per #445 this removes a reference to this._readableState in hash._flush. It was used to get the encoding on the readable side to pass to the writable side but omitting it just causes the stream to handle the encoding issues. PR-URL: #610 Reviewed-By: Chris Dickinson <[email protected]> Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Vladimir Kurchatkin <[email protected]>

Makes streams lazy by default so crypto does not need to monky patch them in order to get good performance. This helps the state objects become part of the private api in nodejs#445.

chrisdickinson · 2015-01-30T21:21:58Z

Cross posting this from nodejs/readable-stream#109:

To amend the intent of this issue a bit – the goal is not to absolutely remove all use of _{{writ,read}able,transform}State, but to remove all places where:

subclasses directly manipulate private state, or...
state is being preserved in the base classes for the exclusive use of a subclass.

Ultimately, the places where the subclasses access those state objects should be catalogued and we should look into the best way to deal with that access – whether that's promotion into a public or protected "reflection" API for subclasses.

Trott · 2016-01-19T06:27:31Z

This issue hasn't seen any activity in almost a year. Is it still active and useful? Or not so much?

indutny · 2016-01-19T08:37:55Z

I think it is very informative, and perhaps needs some work to be done.

Fishrock123 · 2016-06-29T11:13:24Z

Marking as stalled, is this still viable after the problems we encountered last time?

Refs: #445 PR-URL: #34885 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ricky Zhou <[email protected]>

Refs: #445 PR-URL: #34884 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Refs: #445 PR-URL: #34885 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ricky Zhou <[email protected]>

Refs: #445 PR-URL: #34884 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Refs: #445 PR-URL: #34885 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ricky Zhou <[email protected]>

Refs: #445 PR-URL: #34884 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Refs: #445 PR-URL: #34885 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ricky Zhou <[email protected]>

Ref: nodejs#445

Ref: #445 PR-URL: #45063 Refs: #445 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Paolo Insogna <[email protected]> Reviewed-By: Minwoo Jung <[email protected]> Reviewed-By: James M Snell <[email protected]>

RedYetiDev · 2024-09-24T00:12:17Z

(The stalled bot doesn't appear to be working, so I've manually closed. Feel free to reopen if this is no longer stalled)

chrisdickinson mentioned this issue Jan 15, 2015

_debug_agent: use readableObjectMode option for client stream #270

Closed

vkurchatkin mentioned this issue Jan 15, 2015

stream: add isFlowing method #451

Closed

This was referenced Jan 15, 2015

src: nix stdin _readableState.reading manipulation #454

Closed

Cleanup stream state in net #465

Closed

vkurchatkin mentioned this issue Jan 19, 2015

child_process: remove redundant condition #511

Merged

trevnorris added the stream Issues and PRs related to the stream subsystem. label Jan 22, 2015

vkurchatkin mentioned this issue Jan 25, 2015

Release and versioning strategy nodejs/readable-stream#101

Closed

calvinmetcalf mentioned this issue Jan 26, 2015

crypto: remove use of this._readableState #610

Closed

calvinmetcalf mentioned this issue Jan 29, 2015

crypto streams nodejs/readable-stream#109

Closed

calvinmetcalf mentioned this issue Jan 30, 2015

streams: make streams lazy by default #673

Closed

robertrossmann mentioned this issue Mar 25, 2015

_readableState is private, use byteCount lukasredev/skipper-azure#2

Closed

brendanashworth mentioned this issue Aug 27, 2015

net: cleans up 4 instances of internal stream access #2570

Closed

brendanashworth added good first issue Issues that are suitable for first-time contributors. and removed good first issue Issues that are suitable for first-time contributors. labels Aug 29, 2015

awpr mentioned this issue Feb 2, 2016

Lost input from child processes due to initially-flowing net.Socket. #5034

Closed

Fishrock123 added the stalled Issues and PRs that are stalled. label Jun 29, 2016

Trott mentioned this issue May 13, 2021

doc: missing stream.Readable.ReadableState and stream.Writable.WritableState #38678

Closed

1 task

vishnureddy17 mentioned this issue Mar 30, 2022

[question] Question on cleaning up write callbacks in case of error moscajs/aedes#726

Open

Trott added a commit to Trott/io.js that referenced this issue Oct 19, 2022

net: remove _readableStream from debug statement

6007247

Ref: nodejs#445

Trott mentioned this issue Oct 19, 2022

net: remove _readableState from debug statement #45063

Merged

Trott added a commit to Trott/io.js that referenced this issue Oct 19, 2022

net: remove _readableState from debug statement

7bfc179

Ref: nodejs#445

This comment was marked as off-topic.

Sign in to view

sanduluca mentioned this issue Aug 30, 2024

fix(browser): prevent error stream.push() after EOF mqttjs/MQTT.js#1932

Open

RedYetiDev closed this as not planned Won't fix, can't repro, duplicate, stale Sep 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup `_writableState` and `_readableState` access across codebase #445

Cleanup `_writableState` and `_readableState` access across codebase #445

vkurchatkin commented Jan 15, 2015 •

edited by Trott

Loading

chrisdickinson commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

vkurchatkin commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

vkurchatkin commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

chrisdickinson commented Jan 30, 2015

Trott commented Jan 19, 2016

indutny commented Jan 19, 2016

Fishrock123 commented Jun 29, 2016

This comment was marked as off-topic.

RedYetiDev commented Sep 24, 2024 •

edited

Loading

Cleanup _writableState and _readableState access across codebase #445

Cleanup _writableState and _readableState access across codebase #445

Comments

vkurchatkin commented Jan 15, 2015 • edited by Trott Loading

src/node.js

lib/_debug_agent.js

lib/_http_server.js

lib/_tls_legacy.js

lib/_tls_wrap.js

lib/child_process.js

lib/crypto.js

lib/fs.js

lib/net.js

lib/zlib.js

_readableState

_writableState

chrisdickinson commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

vkurchatkin commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

vkurchatkin commented Jan 15, 2015

chrisdickinson commented Jan 15, 2015

chrisdickinson commented Jan 30, 2015

Trott commented Jan 19, 2016

indutny commented Jan 19, 2016

Fishrock123 commented Jun 29, 2016

This comment was marked as off-topic.

RedYetiDev commented Sep 24, 2024 • edited Loading

Cleanup `_writableState` and `_readableState` access across codebase #445

Cleanup `_writableState` and `_readableState` access across codebase #445

vkurchatkin commented Jan 15, 2015 •

edited by Trott

Loading

`src/node.js`

`lib/_debug_agent.js`

`lib/_http_server.js`

`lib/_tls_legacy.js`

`lib/_tls_wrap.js`

`lib/child_process.js`

`lib/crypto.js`

`lib/fs.js`

`lib/net.js`

`lib/zlib.js`

`_readableState`

`_writableState`

RedYetiDev commented Sep 24, 2024 •

edited

Loading