Cannot set property 'parsingHeadersStart' of undefined on 10.15.2

[bjjenson]

• Version:
  10.15.2
• Platform:
  Ubuntu 18.04
• Subsystem:

Out of the blue all of my build servers started to crash today. Each time a request comes in we receive the following error.
Node crashes.

TypeError: Cannot set property 'parsingHeadersStart' of undefined
2019-02-28T22:47:11.137079+00:00 app[web.1]:     at IncomingMessage.resetHeadersTimeoutOnReqEnd (_http_server.js:744:32)
2019-02-28T22:47:11.137082+00:00 app[web.1]:     at IncomingMessage.emit (events.js:194:15)
2019-02-28T22:47:11.137085+00:00 app[web.1]:     at IncomingMessage.EventEmitter.emit (domain.js:441:20)
2019-02-28T22:47:11.137087+00:00 app[web.1]:     at endReadableNT (_stream_readable.js:1103:12)
2019-02-28T22:47:11.137089+00:00 app[web.1]:     at process._tickCallback (internal/process/next_tick.js:63:19)

This is related to version 10.15.2.
After rolling back to 10.15.1 we do not have any more issues.

[rvagg]

@mcollina @nodejs/http

[mcollina]

I’ll send a PR in tomorrow morning.

[mcollina]

Can you please include a repro if you can?

[mcollina]

I've investigated this issue, and I cannot understand when this can happen: we are only setting the parser to null. Protecting this is easy to do:

node/lib/_http_server.js

Line 758 in fedc31b

if (parser !== null) {

this check needs to be extended to undefined. However I don't understand in what conditions it can be undefined and write a specific unit test.

[mcollina]

@rvagg @sam-github @jasnell @addaleax @apapirovski @mscdex do you have any idea?

[jasnell]

Without a repro, no I have no idea

[sam-github]

No idea, either 😕

[bjjenson]

I've tried to repro this locally and cannot. 😕
This happens on both of our Heroku environments however (prod and test).
The steps to repro it there for us was simply to deploy then send a request. It did not matter which request. We got this error every time. By simply reverting to 10.15.1 we were able to get our production server back up and running.
We deployed the code the was running successfully the previous day and it still failed. The only factor was the new release.

Is there anything else that I might be able to provide as insight?

[mcollina]

Are you using any framework? Which ones?

Can you run Node with NODE_DEBUG=http and include a few ~100 lines before the error? It might give us a clue.

[jmorrell]

Can you run Node with NODE_DEBUG=http and include a few ~100 lines before the error? It might give us a clue.

@bjjenson On Heroku, you can do this by running heroku config:set NODE_DEBUG=http -a $APP. I would suggest doing this in a staging environment.

[jasine]

same issue in docker containers

[mcollina]

@jasine can you provide a repro for this? Even the output of a run with NODE_DEBUG=http might be enough, or at least help.

[jasine]

@mcollina run with NODE_DEBUG=http

HTTP 62: write ret = true
HTTP 62: outgoing message end.
HTTP 62: resetHeadersTimeoutOnReqEnd
HTTP 62: AGENT incoming response!
HTTP 62: AGENT socket keep-alive
HTTP 62: CLIENT socket onFree
HTTP 62: agent.on(free) x.x.x.x:80:
HTTP 62: removeSocket x.x.x.x: writable: true
HTTP 62: write ret = true
HTTP 62: outgoing message end.
HTTP 62: SERVER new http connection
HTTP 62: write ret = true
HTTP 62: outgoing message end.
HTTP 62: SERVER socketOnParserExecute 491
HTTP 62: resetHeadersTimeoutOnReqEnd
2019-03-02 17:30:07,824 ERROR 62 nodejs.TypeError: Cannot set property 'parsingHeadersStart' of undefined (uncaughtException throw 1 times on pid:62)
    at IncomingMessage.resetHeadersTimeoutOnReqEnd (_http_server.js:750:32)
    at IncomingMessage.emit (events.js:197:13)
    at endReadableNT (_stream_readable.js:1129:12)
    at processTicksAndRejections (internal/process/next_tick.js:76:17)

pid: 62
hostname: VM-1-13-ubuntu

[Trott]

Code to reproduce:

'use strict';

const http = require('http');

const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('okay', () => { delete res.socket.parser });
  res.end();
});

server.listen(1337, '127.0.0.1');

const req = http.request({
  port: 1337,
  host: '127.0.0.1',
  method: 'GET',
});

req.end();

[lpinca]

@bjjenson @jasine are you using a web framework on the server? If so which one? Can you give us more details on your environment?

[jasine]

@bjjenson @jasine are you using a web framework on the server? If so which one? Can you give us more details on your environment?

the web framework i use is egg , as far as I know, it is based on koa, all of the http request is ok, the issue happened when the websocket connected, the client and server side websocket library is socket.io

[lpinca]

@jasine what version of socket.io? Run npm ls socket.io. I still can't reproduce with this:

const Koa = require('koa');
const socketio = require('socket.io');
const { createServer } = require('http');

const body = `<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script src="/socket.io/socket.io.js"></script>
    <script>
      (function () {
        const socket = io('http://localhost:3000');;

        socket.on('connect', function() {
          console.log('open');
        });
      })();
    </script>
  </body>
</html>`;

const app = new Koa();

app.use(async function(ctx) {
  ctx.body = body;
});

const server = createServer(app.callback());
const io = socketio(server);

io.on('connection', function() {});

server.listen(3000, function() {
  console.log('Listening on [::]:3000');
});

[wyqsess]

@jasine what version of socket.io? Run npm ls socket.io. I still can't reproduce with this:

const Koa = require('koa');
const socketio = require('socket.io');
const { createServer } = require('http');

const body = `<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script src="/socket.io/socket.io.js"></script>
    <script>
      (function () {
        const socket = io('http://localhost:3000');;

        socket.on('connect', function() {
          console.log('open');
        });
      })();
    </script>
  </body>
</html>`;

const app = new Koa();

app.use(async function(ctx) {
  ctx.body = body;
});

const server = createServer(app.callback());
const io = socketio(server);

io.on('connection', function() {});

server.listen(3000, function() {
  console.log('Listening on [::]:3000');
});

i have the same issue too when try to connect in socket.io

version:
"socket.io": "^2.1.0"
"socket.io-client": "^2.1.1",

And i find that it will work well at [email protected], and crash at [email protected]

[hyj1991]

@mcollina Same issue.

Here is a repro with egg.js: egg-socket.io-error-demo

Node version: v8.15.1 (and v8.15.0 is well...)

Error message:

2019-03-05 14:52:21,167 ERROR 20840 nodejs.TypeError: Cannot set property 'parsingHeadersStart' of undefined (uncaughtException throw 1 times on pid:20840)
    at IncomingMessage.resetHeadersTimeoutOnReqEnd (_http_server.js:728:32)
    at emitNone (events.js:106:13)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

pid: 20840
hostname: hyj1991.local

[mcollina]

#26402 fixes it @hyj1991.

[lpinca]

@hyj1991

The problem is that request.socket is replaced by egg-socket.io with the socket.io Socket. See https://github.com/eggjs/egg-socket.io/blob/9e7f71d835930d3a63c69635488737066f779661/lib/connectionMiddlewareInit.js#L8-L9.

This is not ok and can break again in future.

[snail-boy]

i have a idea ， client 不要用socket-client 换成egg-socket.io 可以