CI failures: 20180925

[joyeecheung]

Failures in node-test-pull-request/17334 to node-test-pull-request/17430 that failed more than 2 PRs
(Generated using ncu-ci walk pr --stats --copy)

JSTest Failure

Reason	parallel/test-bootstrap-modules
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22845, nodejs/node#22951, nodejs/node#23014, nodejs/node#23010, nodejs/node#23042, nodejs/node#23044, nodejs/node#22998)
Appeared	undefined
First CI	https://ci.nodejs.org/job/node-test-pull-request/17339/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17399/

Example

not ok 154 parallel/test-bootstrap-modules
  ---
  duration_ms: 0.336
  severity: fail
  exitcode: 1
  stack: |-
    
    events.js:167
          throw er; // Unhandled 'error' event
          ^
    AssertionError [ERR_ASSERTION]: Internal Binding contextify,Internal Binding worker,NativeModule events,Internal Binding trace_events,NativeModule internal/safe_globals,NativeModule internal/async_hooks,NativeModule internal/errors,Internal Binding uv,Binding buffer,Internal Binding async_wrap,Binding config,Binding icu,NativeModule util,NativeModule internal/validators,NativeModule internal/encoding,NativeModule internal/util,Binding constants,Internal Binding util,NativeModule internal/util/types,Internal Binding types,NativeModule buffer,NativeModule internal/buffer,NativeModule internal/process/per_thread,NativeModule internal/process/worker_thread_only,NativeModule internal/process/stdio,NativeModule internal/worker,NativeModule assert,NativeModule internal/assert,NativeModule fs,NativeModule path,NativeModul...

---

Reason	parallel/test-gc-net-timeout
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22331, nodejs/node#23058, nodejs/node#22846, nodejs/node#22974, nodejs/node#22947, nodejs/node#23056, nodejs/node#23047)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-rackspace-win2012r2-x64-5, test-joyent-smartos16-x64-1, test-osuosl-aix61-ppc64_be-1, test-digitalocean-freebsd11-x64-2, test-digitalocean-ubuntu1804_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17402/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17428/

Example

not ok 1366 parallel/test-gc-net-timeout
  ---
  duration_ms: 120.88
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 500 requests
    Done: 10/500
    Collected: 0/22
    Done: 33/500
    Collected: 10/44
    Done: 55/500
    Collected: 33/66
    Done: 64/500
    Collected: 55/77
    Done: 77/500
    Collected: 64/99
    Done: 110/500
    Collected: 77/121
    Done: 132/500
    Collected: 110/143
    Done: 154/500
    Collected: 132/165
    Done: 187/500
    Collected: 154/187
    Done: 211/500
    Collected: 187/231
    Done: 231/500
    Collected: 211/253
    Done: 253/500
    Collected: 231/275
    Done: 275/500
    Collected: 253/286
    Done: 289/500
    Collected: 275/308
    Done: 321/500
    Collected: 289/341
    Done: 352/500
    Collected: 321/363
    Done: 375/500
    Collected: 352/396
    Done: 407/500
    Collected: 375/418
    Done: 431/500
    Collected: 407/440
    Done: 451/500
    Collected: 431/462
    Done: 455/500
    Collected: 451/484
    Done: 484/500
    Co...

---

Reason	sequential/test-http2-session-timeout
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22953, nodejs/node#22951, nodejs/node#23010, nodejs/node#23048, nodejs/node#22331, nodejs/node#22413, nodejs/node#22914)
Appeared	test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_securogroup-debian9-armv6l_pi1p-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17379/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17409/

Example

not ok 322 sequential/test-http2-session-timeout
  ---
  duration_ms: 6.310
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:126
      throw err;
      ^
    
    AssertionError [ERR_ASSERTION]: function should not have been called at /home/iojs/build/workspace/node-test-binary-arm/test/sequential/test-http2-session-timeout.js:9
        at Http2Server.mustNotCall (/home/iojs/build/workspace/node-test-binary-arm/test/common/index.js:407:12)
        at Http2Server.emit (events.js:182:13)
        at ServerHttp2Session.sessionOnTimeout (internal/http2/core.js:2631:15)
        at Object.onceWrapper (events.js:273:13)
        at ServerHttp2Session.emit (events.js:182:13)
        at ServerHttp2Session._onTimeout (internal/http2/core.js:1306:10)
        at ontimeout (timers.js:454:11)
        at tryOnTimeout (timers.js:326:5)
        at listOnTimeout (timers.js:300:5)
        at processTimers (timers.js:257:5)
  ...

---

Reason	parallel/test-gc-http-client-connaborted
Type	JS_TEST_FAILURE
Failed PR	5 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23048, nodejs/node#23056)
Appeared	test-rackspace-win2012r2-x64-5, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-alpine37_container-x64-2, test-rackspace-win2008r2-x64-6
First CI	https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17430/

Example

not ok 184 parallel/test-gc-http-client-connaborted
  ---
  duration_ms: 120.66
  severity: fail
  exitcode: 1
  stack: |-
    timeout
    We should do 500 requests
    Done: 44/500
    Collected: 0/99
    Done: 131/500
    Collected: 44/176
    Done: 219/500
    Collected: 131/264
    Done: 271/500
    Collected: 219/407
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
   ...

---

Reason	parallel/test-gc-http-client-onerror
Type	JS_TEST_FAILURE
Failed PR	4 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23056)
Appeared	undefined, test-packetnet-centos7-arm64-1, test-digitalocean-alpine37_container-x64-2, test-digitalocean-alpine37_container-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17426/

Example

not ok 1395 parallel/test-gc-http-client-onerror
  ---
  duration_ms: 120.26
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 500 requests
    Done: 0/500
    Collected: 0/40
    Done: 50/500
    Collected: 0/90
    Done: 90/500
    Collected: 40/130
    Done: 140/500
    Collected: 80/180
    Done: 190/500
    Collected: 130/230
    Done: 243/500
    Collected: 180/290
    Done: 290/500
    Collected: 240/330
    Done: 335/500
    Collected: 280/380
    Done: 390/500
    Collected: 330/430
    Done: 417/500
    Collected: 380/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 4...

---

Reason	parallel/test-gc-http-client-timeout
Type	JS_TEST_FAILURE
Failed PR	4 (nodejs/node#23058, nodejs/node#22906, nodejs/node#22947, nodejs/node#23075)
Appeared	test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2, test-digitalocean-ubuntu1804_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17403/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17420/

Example

not ok 1186 parallel/test-gc-http-client-timeout
  ---
  duration_ms: 120.91
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 550 requests
    Done: 0/550
    Collected: 0/33
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    Done: 0/550
    Collected: 0/44
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    ti...

---

Reason	parallel/test-gc-http-client
Type	JS_TEST_FAILURE
Failed PR	3 (nodejs/node#23072, nodejs/node#23048, nodejs/node#23056)
Appeared	test-rackspace-win2012r2-x64-4, test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17417/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17425/

Example

not ok 184 parallel/test-gc-http-client
  ---
  duration_ms: 120.81
  severity: fail
  exitcode: 1
  stack: |-
    timeout
    We should do 300 requests
    Done: 29/300
    Collected: 0/90
    Done: 59/300
    Collected: 29/120
    Done: 90/300
    Collected: 59/150
    Done: 104/300
    Collected: 90/180
    Done: 104/300
    Collected: 90/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
  ...

---

Reason	parallel/test-crypto-keygen
Type	JS_TEST_FAILURE
Failed PR	2 (nodejs/node#22845, nodejs/node#22969)
Appeared	test-packetnet-centos7-arm64-1, test-linuxonecc-rhel72-s390x-3, test-azure_msft-win10-x64-3
First CI	https://ci.nodejs.org/job/node-test-pull-request/17341/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17343/

Example

not ok 2388 parallel/test-crypto-keygen
  ---
  duration_ms: 240.71
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    assert.js:662
        throw actual;
        ^
    
    Error: error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
        at Sign.sign (internal/crypto/sig.js:83:27)
        at testSignVerify (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:43:42)
        at assert.throws (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:171:7)
        at getActual (assert.js:574:5)
        at Function.throws (assert.js:692:24)
        at AsyncWrap.common.mustCall (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:170:12)
        at AsyncWrap.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/common/index.js:350:15)
        at AsyncWrap.wrap.ondone (internal/crypto/...

---

Reason	parallel/test-net-connect-options-port
Type	JS_TEST_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22846)
Appeared	test-packetnet-ubuntu1604-arm64-2, test-digitalocean-alpine37_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17407/

Example

not ok 2361 parallel/test-net-connect-options-port
  ---
  duration_ms: 240.140
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

---

Build Failure

Reason	sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory
Type	BUILD_FAILURE
Failed PR	3 (nodejs/node#22891, nodejs/node#22969, nodejs/node#20713)
Appeared	test-linuxonecc-rhel72-s390x-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17334/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17336/

Example

sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory

---

Reason	ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22914)
Appeared	test-requireio_davglass-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-arm64_pi3-2, test-requireio_pivotalagency-debian9-arm64_pi3-2, test-requireio_williamkapke-debian9-arm64_pi3-3, test-requireio_kahwee-debian9-arm64_pi3-1, test-requireio_notthetup_sayanee-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-armv6l_pi1p-2, test-requireio_securogroup-debian9-arm64_pi3-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17427/

Example

ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/iojs/build/workspace/node-test-binary-arm/out/junit/test.xml is 29 min old

Notifying upstream projects of job completion
Finished: FAILURE

---

Reason	error: pathspec ... did not match any file(s) known to git.
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22969, nodejs/node#22418)
Appeared	test-azure_msft-win10-x64-3, test-azure_msft-win10-x64-4, test-rackspace-win2008r2-x64-2, test-rackspace-win2008r2-x64-1, test-azure_msft-win2008r2-x64-1, test-azure_msft-win2008r2-x64-2, test-rackspace-win2012r2-x64-9, test-azure_msft-win2012r2-x64-2, test-rackspace-win2012r2-x64-4, test-rackspace-win2012r2-x64-2, test-rackspace-win2012r2-x64-5, test-azure_msft-win2012r2-x64-1, test-rackspace-win2012r2-x64-3, test-rackspace-win2012r2-x64-8, test-azure_msft-win2016-x64-2, test-azure_msft-win2016-x64-5, test-azure_msft-win2016-x64-3, test-azure_msft-win2016-x64-4, test-azure_msft-win10-x64-2, test-rackspace-win2008r2-x64-5, test-rackspace-win2012r2-x64-7, test-rackspace-win2012r2-x64-6
First CI	https://ci.nodejs.org/job/node-test-pull-request/17347/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17359/

Example

error: pathspec 'refs/remotes/jenkins_tmp/jenkins-node-test-commit-windows-fanned-94402b043f54b82c706b86f02d05b5290e48c872-bin-win-vcbt2015' did not match any file(s) known to git.

c:\workspace\node-test-binary-windows>if 1 NEQ 0 exit /b 1 
Build step 'Execute Windows batch command' marked build as failure
Run condition [Always] enabling perform for step [[Execute Windows batch command, Execute shell]]
[node-test-binary-windows] $ cmd /c call C:\Users\ADMINI~1\AppData\Local\Temp\jenkins984036347200566261.bat

---

Reason	fatal: Not a git repository (or any of the parent directories): .git
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22950, nodejs/node#23004)
Appeared	test-rackspace-win2008r2-x64-4
First CI	https://ci.nodejs.org/job/node-test-pull-request/17357/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17358/

Example

fatal: Not a git repository (or any of the parent directories): .git

---

Reason	sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22914)
Appeared	test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_svincent-debian9-armv7l_pi2-1, test-requireio_mininodes-debian9-armv6l_pi1p-1, test-requireio_mcollina-debian9-armv7l_pi2-1, test-requireio_securogroup-debian9-armv6l_pi1p-1, test-requireio_rvagg-debian9-armv7l_pi2-1, test-requireio_bengl-debian9-armv6l_pi1p-2, test-requireio_joeyvandijk-debian9-armv7l_pi2-2, test-requireio_indieisaconcept-debian9-armv6l_pi1p-1, test-requireio_joeyvandijk-debian9-armv7l_pi2-1, test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_jasnell-debian9-armv7l_pi2-1, test-requireio_ceejbot-debian9-armv7l_pi2-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17427/

Example

sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

---

Git Failure

Jenkins Failure

Reason	java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
Type	JENKINS_FAILURE
Failed PR	2 (nodejs/node#22997, nodejs/node#22998)
Appeared	test-macstadium-macos10.11-x64-2, test-rackspace-win2008r2-x64-4
First CI	https://ci.nodejs.org/job/node-test-pull-request/17356/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17399/

Example

java.io.IOException: Backing channel 'JNLP4-connect connection from 207.254.58.162/207.254.58.162:62697' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
	at com.sun.proxy.$Proxy85.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)

[joyeecheung]

Progress

• java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
• parallel/test-net-connect-options-port

In progress

• sequential/test-http2-session-timeout: trying to improve in Minor improvements for test-http2-session-timeout node#23058
• parallel/test-gc-net-timeout: Investigate flaky test-gc-net-timeout node#23067 , trying to improve in test: improve reliability of test-gc-net-timeout node#23086
• parallel/test-gc-http-client-connaborted: trying to improve in test: improve test-gc-http-client-connaborted node#23091
• parallel/test-gc-http-client-onerror: Investigate flaky test-gc-http-client-onerror node#23089 trying to improve in test: improve test-gc-http-client-onerror node#23090
• parallel/test-gc-http-client-timeout: Investigate flaky test-gc-http-client-timeout node#23066 trying to improve in test: improve test-gc-http-client-timeout node#23088
• parallel/test-gc-http-client: Investigate flaky parallel/test-gc-http-client node#22336 trying to improve in test: improve reliability of test-gc-http-client node#23087

Fixed

• parallel/test-crypto-keygen: Investigate flaky parallel/test-crypto-keygen node#22978 fixed in test: fix flaky key pair generation test node#22980
• parallel/test-bootstrap-modules: lib: lazy load internal/queue_microtask node#23046
• sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory: node-test-commit-linuxone cannot find ./build/jenkins/scripts/select-compiler.sh build#1501

Resume failure

Tracking in nodejs/build#1496

• fatal: Not a git repository (or any of the parent directories): .git
• error: pathspec ... did not match any file(s) known to git.
• ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
• sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

[richardlau]

sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory was addressed by nodejs/build#1501.

[refack]

parallel/test-gc-http-client - nodejs/node#22336
parallel/test-gc-http-client-timeout - nodejs/node#23066
parallel/test-gc-net-timeout - nodejs/node#23067

@joyeecheung could you update the pattern matching to ignore
sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
and
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
Those are errors that don't cause build FAILURE. In those cases the error is probably something else. In the case of node-test-binary-arm it usually means that a resume build failed because the temporary branch passing the binaries was deleted during the original build.

[joyeecheung]

@refack I can add an ad-hoc pattern for now (maybe replace/append that line with (possible resume failures)), yes, but I am not sure how long that association will hold? (Maybe I can just add a TODO in the tool and lift it when there is a new association)

[refack]

@joyeecheung What I meant is that in the current configuration of CI
sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
and
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
should not cause the build to fail.

You are right about associating them with a real cause, "failed resume" is just a guess.

[joyeecheung]

@refack I see. The reasons here are not accurate reasons, they are more like keys to group the errors by.
Just think of another solution: I can check if the PR build comes from another PR build (i.e. a resumed build) and check if the failed job is a fanned one. If so I'll tentatively mark them as RESUME_FAILURE (a new category). That's how my brain currently works anyway.

[refack]

From a big picture POV, I would like to tip my 🎩 to @joyeecheung @Trott @joaocgreis and rest of @nodejs/build that (AFAIR for the first time ) made it so infra failures do not dominate this list. One small step...

[Trott]

@refack Right back at you times 💯