add support for running tests in parallel

[boneskull]

(this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR)

Summary

This PR adds support for running test files in parallel via --parallel. For many cases, this should "just work."

How It Works

When the --parallel flag is supplied, Mocha will swap out the default Runner (lib/runner.js) for BufferedRunner (lib/buffered-runner.js).

BufferedRunner extends Runner. BufferedRunner#run() is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker threads) based on the maximum job count (--jobs; defaults to <number of CPU cores> - 1). Both BufferedRunner and the worker module consume the abstraction layer, workerpool.

BufferedRunner#run() does not load the test files, unlike Runner#run(). Instead, it has a list of test files, and puts these into an async queue. The EVENT_RUN_BEGIN event is then emitted. As files enter the queue, BufferedRunner#run() tells workerpool to execute the run() function of the pool. workerpool then launches as many worker processes are needed--up to the maximum--and executes the run() function with a single filepath and any options for a Mocha instance.

The first time lib/worker.js is invoked, it will "bootstrap" itself, by handling --require'd modules and validating the UI. Note that reporter validation does not occur. Once bootstrapped, it instantiate Mocha, add the single file, swap any reporter out for the Buffered reporter (lib/reporters/buffered.js) then execute Mocha#run(), which invokes Runner#run().

The Buffered reporter listens for events emitting from the Runner instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. BufferedRunner#run() receives the list of events, trivially deserializes them, and re-emits the events to whatever the chosen reporter is (e.g., the spec reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file.

Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in BufferedRunner#run()'s queue, workerpool will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the EVENT_RUN_END event is emitted, and the reporter handles it.

Limitations & Caveats

(this section is pasted from the documentation with minimal edits)

Reporter Limitations

Due to the nature of the following reporters, they cannot work when running tests in parallel:

• markdown
• progress
• json-stream

These reporters expect Mocha to know how many tests it plans to run before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run.

In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is buffered; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical).

Order is Non-Deterministic

In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files.

Because of this, the following options cannot be used in parallel mode:

• --file
• --sort
• --delay

Test Duration Variability

Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise.

"Bail" is "Best Effort"

When used with --bail (or this.bail()) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting.

Likewise, subprocesses may throw uncaught exceptions. When used with --allow-uncaught, Mocha will "bubble" this exception to the main process, but still must shut down its processes.

Root Hooks Are Not Global

NOTE: This only applies to test files run parallel mode.

A root-level hook is a hook in a test file which is not defined within a suite. An example using the bdd interface:

// test/setup.js
beforeEach(function() {
  doMySetup();
});

afterEach(function() {
  doMyTeardown();
});

When run (in the default "serial" mode) via mocha --file "./test/setup.js" "./test/**/*.spec.js", setup.js will be executed first, and install the two hooks shown above for every test found in ./test/**/*.spec.js.

When Mocha runs in parallel mode, test files do not share the same process. Consequently, a root-level hook defined in test file A won't be present in test file B.

There are a (minimum of) two workarounds for this:

1. require('./setup.js') or import './setup.js' at the top of every test file. Best avoided for those averse to boilerplate.
2. Recommended: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system.

No Browser Support

Parallel mode is only available in Node.js.

Migration Checklist

If you find your tests don't work properly when run with --parallel, either shrug and move on, or use this handy-dandy checklist to get things working:

• ✅ Ensure you are using a supported reporter.
• ✅ Ensure you are not using other unsupported flags.
• ✅ Double-check your config file; options set in config files will be merged with any command-line option.
• ✅ Look for root-level hooks in your tests. Move them into a root hook plugin.
• ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode.
• ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via --timeout)
• ✅ Ensure your tests do not depend on being run in a specific order.
• ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files.

Caveats About Testing in Parallel

Some types of tests are not so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.).

Free-tier butt CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a .mocharc.js to check for process.env.CI, and adjust the job count as appropriate.

It's unlikely (but not impossible) to see a performance gain from a job count greater than the number of available CPU cores. That said, play around with the job count--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster!

Changeset Notes

• updated signal handling in bin/mocha to a) better work with Windows, and b) work properly with --parallel to avoid leaving zombie workers
• docstrings in lib/cli/collect-files.js
• refactors in lib/cli/run-helpers.js and lib/cli/watch-run.js. We now have four methods:
  • watchRun() - serial + watch
  • singleRun() - serial
  • parallelWatchRun() - parallel + watch
  • parallelRun() - parallel
• lib/cli/run.js and lib/cli/run-option-metadata.js: additions for new options and checks for incompatibility
• add lib/reporters/buffered.js (Buffered); this reporter is not re-exported in Mocha.reporters, since it should only be invoked internally.
• tweak landing reporter to avoid referencing Runner#total, which is incompatible with parallel mode. It didn't need to do so in the first place!
• the tap reporter now outputs the plan at the end instead of at the beginning (avoiding a call to Runner#grepTotal(), which is incompatible with parallel mode). This is within spec, so should not be a breaking change.
• add lib/buffered-runner.js (BufferedRunner); subclass of Runner which overrides the run() method.
  • There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere.
  • when DEBUG=mocha:parallel* is in the env, this module will output statistics about the worker pool every 5s
  • the run() method looks a little weird because I wanted to use async/await, but the method it is overriding (Runner#run) is not async
  • traps SIGINT to gracefully terminate the pool
  • pulls in promise.allsettled polyfill to handle workers that may have rejected with uncaught exceptions
  • "bail" support is best-effort.
  • the ABORTING state is only for interruption via SIGINT or if allowUncaught is true and we get an uncaught exception
• Hook, Suite, Test: add a serialize() method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its Runner; event arguments (e.g., test or suite) are serialized in this manner. Note that this limits what reporters have access to, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The $$ convention tells the deserializer to turn the property into a function which returns the passed value, e.g., test.fullTitle().
• lib/mocha.js:
  • refactor Mocha#reporter for nicer parameter & variable names
  • rename loadAsync to lazyLoadFiles, which is more descriptive, IMO. It's a private property, so should not be a breaking change.
  • Constructor will dynamically choose the appropriate Runner
• lib/runner.js: BufferedRunner needs the options from Mocha#options, so I updated the parent method to define the parameter. It is unused here.
• add lib/serializer.js: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process.
  • I spent a long time trying to get this working. We need to account for things like Error instances, with their stack traces, since those can be event arguments (e.g., EVENT_TEST_FAIL sends both a Test and the Error). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well.
  • Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures.
  • Attempted to avoid prototype pollution issues
  • Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff.
  • The __type prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into Error instances, for example. You can't send an Error instance over IPC!
• add lib/worker.js:
  • registers its run() function with workerpool to be called by main process
  • if DEBUG=mocha:parallel* is set, will output information (on an interval) about long-running test files
  • afaik the only way run() can reject is if allowUncaught is true or serialization fails
  • any user-supplied reporter value is replaced with the Buffered reporter. thus, reporters are not validated.
  • the worker uses Runner, like usual.
• tests:
  • see test/integration/options/parallel.spec.js for the interesting stuff
  • upgrade unexpected for "to have readonly property" assertion
  • upgrade unexpected-eventemitter for support async function support
  • integration test helpers allow Mocha's developers to use --bail and --parallel, but will default to --no-bail and --no-parallel.
• etc:
  • update .eslintrc.yml for new Node-only files
  • increase default timeout to 1000 (also seen in another PR) and use parallel mode by default in .mocharc.yml
  • run node unit tests in serial as sort of a smoke test, as otherwise all our tests would be run in parallel
  • karma, browserify: ignore files for parallel support
  • force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color

Ref: #4198

[boneskull]

In general, I believe this to be a semver minor change, but it might want to land in v8.0.0 anyway.

[boneskull]

updated package-lock.json

[boneskull]

This needs changes from #4240 and #4244 in order for the build to pass.

[boneskull]

--grep doesn't work because the value winds up being a RegExp, and you can't express such a thing in JSON, so I'm going to need to add something that serializes/deserializes options, or we coerce the value to a regexp just-in-time.

will also need to see if there are other non-primitive-non-array options that might run into trouble

also, I don't think exclusive tests can work. we can certainly skip tests, but since we aren't loading all files up-front, we won't necessarily be able to tell if there's an only somewhere until after we've already run other tests...

[coveralls]

Coverage increased (+0.2%) to 93.585% when pulling 2ed47ef on boneskull/issue/2839-parallel into 6d60eb0 on master.

[boneskull]

#4244 and #4240 have both been merged, and this has been rebased onto master. It's ready for review, assuming CI passes.

[Munter]

workerpool looks browser compatible enough to support our browser range. Is there a way to actually get these files in the browser bundle and get parallel runs in there as well?

I'm not super happy with keeping this blacklist for browser builds. It would be better if we could make a split where they are actually separate dependency graphs to make it less magic. I've already started doing some of that on the rollup branch, so I might just follow up on that thread there or later after both of those features are merged

[boneskull]

workerpool looks browser compatible enough to support our browser range. Is there a way to actually get these files in the browser bundle and get parallel runs in there as well?

yes, it might work. I didn't want to try it in this PR though.

[nicojs]

I've taken a look. You nailed the serialization stuff, couldn't really find anything wrong with it.

I've got quite a few remarks, but some of them are questions and others are opinionated. Please tell me if you disagree with them.

[nicojs]

Why is it called a "Buffered" runner? Since the description isn't explaining that either. The fact that it uses a buffer internally doesn't help the consumer of this class.

Suggestion: ParallelRunner.

Also: we're allowed to use es classes? Awesome! 😍

[boneskull]

Why is it called a "Buffered" runner? Since the description isn't explaining that either. The fact that it uses a buffer internally doesn't help the consumer of this class.

BufferedParallelRunner could be more accurate. The idea is that ParallelRunner is fine until we add an unbuffered parallel runner; e.g., a streaming parallel runner. This would be especially useful for streaming machine-readable results; not so much for human-readable output. It would more closely mimic serial mode; how reporters generate output in real-time. But it would also be confusing to use with a structured reporter like spec, because it'd just be a firehose with random indentation. Would work for dot, though.

Also: we're allowed to use es classes? Awesome! 😍

This is fair game in bin/, scripts/, lib/nodejs/, lib/cli/ and test/node-unit... essentially, code that only runs in Node.js.

[nicojs]

BufferedParallelRunner could be more accurate. The idea is that ParallelRunner is fine until we add an unbuffered parallel runner; e.g., a streaming parallel runner.

I would expect the streaming might be a setting of the ParellelRunner in the future. It is a minor implementation detail, right? Just a streaming reporter configured in the worker process, rather than a buffered reporter.

[nicojs]

Why the need for $$? We're using 2 'magic' serializer markers. __type and $$-prefixed values. Maybe we can align those to be the same. I would also like the purpose to be directly clear for readers, maybe prefix with: __serialized__?

[boneskull]

these magic keys do have two different meanings, but I agree, I can be much more explicit about their meaning.

[boneskull]

I am going to defer this task until later. I think there are more refactors that can happen here, but my goal is to get this merged first. not too concerned about regressions

[nicojs]

Do you feel it's part of an MVP? Or can this land & be released without it?

Unsure if a joke 😏. I don't think anyone could argue that progress reporting is a must-have for MVP 😉. People can opt-out of --parallel mode if they feel strongly about it.

[boneskull]

imo this is useful just to leave in.

[boneskull]

my assumption is that people tend not to love nested ternaries. but if someone wants it, I will abide

[boneskull]

@nicojs Was thinking about the lazyLoadFiles thing, and the variable is not really needed. What's going on is that, by default, Mocha#run will implicitly call Mocha.loadFiles unless this variable is set. Perhaps it's better to load files explicitly; rather than:

mocha.addFile('my-file.js');
mocha.run()

do:

mocha.addFile('my-file.js');
mocha.loadFiles();
mocha.run();

For the parallel/ESM case, it's currently:

mocha.addFile('my-file.js');
mocha.lazyLoadFiles = false;
await mocha.loadFilesAsync();
mocha.run();

but it would then eliminate the need to set the variable:

mocha.addFile('my-file.js');
await mocha.loadFilesAsync();
mocha.run();

This would be a breaking change, however, and so might not be worth doing. But it also allows us to add a state between INIT and RUNNING, e.g., READY, which is set to true once one of Mocha#loadFiles or Mocha#loadFilesAsync has successfully completed. It complicates the state machine, so I didn't want to do it... but figure it's worth mentioning.

[nicojs]

Note: the state of the mocha instance is reset to INIT after a run (if cleanReferencesAfterRun is false). So a user can change the parallelMode after mocha has run.

This is fine, as long as we implement the dispose on the bufferedRunner correctly. It would mean that the parallel runner / buffered runner is disposed and a new one is started. We might want to change this behavior in the future, but it's fine for now I would say.

[boneskull]

appveyor is acting very strangely; I'm not sure why a worker thread is getting invoked in the "watch" tests. I've squashed this again

[boneskull]

@nicojs I think I've addressed your concerns here, but at this point I'm unsure because I'm not sure where to find your review.

I currently can't view the Netlify deploy to see what's wrong there, but, the docs build on my machine. 😄

[nicojs]

Thanks @boneskull . I will try to get to this today. Might be tomorrow.

[nicojs]

@boneskull I think this can land as is. I would prefer to get the name change in (BufferedRunner -> ParallelBufferedRunner), but otherwise this looks good.

[kyrylodolynskyi]

When do you plan to publish this feature?