test: make crypto.timingSafeEqual test less flaky #8456
Conversation
|
Haven't been able (yet) to force failures via stress test repetition but have seen it coming up on full CI runs, so here's three of those: CI: https://ci.nodejs.org/job/node-test-pull-request/3975/ CI again: https://ci.nodejs.org/job/node-test-pull-request/3976/ CI one more time: https://ci.nodejs.org/job/node-test-pull-request/3977/ |
| const crypto = require('crypto'); | ||
|
|
||
| const BENCHMARK_FUNC_PATH = | ||
| '../fixtures/crypto-timing-safe-equal-benchmark-func'; |
There was a problem hiding this comment.
Nit: common.fixturesDir but probably no need to worry about that until we see if this even works.
|
Stress test on debian8-x86 with master shows a 30% failure rate. https://ci.nodejs.org/job/node-stress-single-test/894/nodes=debian8-x86/console
|
|
Timeout on Raspberry Pi is probably acceptable. A few other tests do this to skip tests on machines with low RAM (like the old Pi devices): if (!common.enoughTestMem) {
common.skip(skipMessage);
return;
}We can do that on this test too if the Raspberry Pi 1 can't handle it. |
|
I think this stresstest is running the wrong test; it should be |
|
Whoops, yes, let's re-run that stress test but with the correct test this time: https://ci.nodejs.org/job/node-stress-single-test/nodes=debian8-x86/898/console |
|
I don't want to monopolize our only debian8-x86 test machine for 14 hours (which is what I'm calculating it would take to run this test 10K times... 10K * 5 seconds = 50K seconds = ~14 hours). So I'm going to stop it now after ~80 runs without a single failure. Certainly more than enough to feel confident that the failure rate on debian8-x86 will be much less than the 30% we are seeing on current master. Maybe add the skip code and then we can run regular CI a few more times to see if everything is A-OK? |
| } | ||
|
|
||
| if (!common.enoughTestMem) { | ||
| common.skip('skipping memory-intensive test'); |
There was a problem hiding this comment.
Nit: common.skip() prepends text indicating the test has been skipped so the word 'skipping' is redundant. I'm sure this is actually an issue with a bunch of other tests and is fine to leave as-is (hence the 'Nit' prefix) if you wish, as fixing these throughout the tests would probably make a good first contribution for a newcomer anyway.
There was a problem hiding this comment.
Based on a search in the test/ folder it seems like all the other tests actually do this correctly, so I fixed this one to do it correctly as well.
not-an-aardvark
changed the title
|
CI looks great. (Windows failure looks unrelated, will open separate issue to investigate if one isn't already open.) @nodejs/testing @nodejs/crypto |
|
LGTM |
The `crypto.timingSafeEqual` test still seems to be a bit flaky. This makes a few changes to the test: * Separates the basic usage and the benchmarking into different tests * Moves the timing-sensitive benchmark function into a separate module, and reparses the module on every iteration of the loop to avoid shared state between timing measurements. PR-URL: nodejs#8456 Reviewed-By: James M Snell <[email protected]>
|
Landed in c678ecb 🎉 |
The `crypto.timingSafeEqual` test still seems to be a bit flaky. This makes a few changes to the test: * Separates the basic usage and the benchmarking into different tests * Moves the timing-sensitive benchmark function into a separate module, and reparses the module on every iteration of the loop to avoid shared state between timing measurements. PR-URL: #8456 Reviewed-By: James M Snell <[email protected]>
Checklist
make -j4 test(UNIX), orvcbuild test nosign(Windows) passesAffected core subsystem(s)
crypto
Description of change
WIP; do not merge.The
crypto.timingSafeEqualtest still seems to be a bit flaky. This makes a few changes to the test:If this doesn't work, an alternative would be to start a separate child process for each individual timing measurement, which would completely avoid shared state between measurements (although it would also probably make the test much more CPU-intensive).
/cc @Trott
Refs: #8040, #8203, #8304