test_runner: add support for coverage thresholds

[OnlyAviv]

Fixes #48739

This PR adds support for coverage thresholds.

The following new CLI flags are added:

• --test-coverage-branches
• --test-coverage-functions
• --test-coverage-lines

---

Notable Change

Node.js now supports requiring code coverage to meet a specific threshold before the process exits successfully. To use this feature, you need to enable the --experimental-test-coverage flag.

You can set thresholds for the following types of coverage:

• Branch coverage: Use --test-coverage-branches=<threshold>
• Function coverage: Use --test-coverage-functions=<threshold>
• Line coverage: Use --test-coverage-lines=<threshold>

<threshold> should be an integer between 0 and 100. If an invalid value is provided, a TypeError will be thrown.

If the code coverage fails to meet the specified thresholds for any category, the process will exit with code 1.

For instance, to enforce a minimum of 80% line coverage and 60% branch coverage, you can run:

$ node --experimental-test-coverage --test-coverage-lines=80 --test-coverage-branches=60 example.js

# start of coverage report
# ...
# ...
# ...
# end of coverage report
# Error: 53.45% line coverage does not meet the threshold of 80%.
# Error: 40.30% branch coverage does not meet the threshold of 60%.

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[OnlyAviv]

To reviewers, please send as any changes as you'd like. This is still a draft, so I'd love lots and lots of feedback

[ljharb]

there's always 4 thresholds: branch, function, line, and statements. can statements be added also?

[OnlyAviv]

there's always 4 thresholds: branch, function, line, and statements. can statements be added also?

I don't believe the Node.js coverage reporter handlers statements

[OnlyAviv]

I realize that this is passing local tests, and is not 'incomplete' per say, so it doesn't hurt to keep it in the review state.

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @RedYetiDev.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[marco-ippolito]

I gave it a try in the past #51370 and it was blocked

[OnlyAviv]

I gave it a try in the past #51370 and it was blocked

Looking at yours, it looks like a much smarter way to do this than I did. If that didn't land, I doubt this will.

[mcollina]

I was ok then and I'm ok now, but yeah, this should probably not land unless someone wants to move it through a @nodejs/tsc vote.

[OnlyAviv]

I was ok then and I'm ok now, but yeah, this should probably not land unless someone wants to move it through a @nodejs/tsc vote.

+1 with a TSC vote. If the consensus is to ship this feature, I think @marco-ippolito's PR is a better layout of this.

[marco-ippolito]

I was ok then and I'm ok now, but yeah, this should probably not land unless someone wants to move it through a @nodejs/tsc vote.

+1 with a TSC vote. If the consensus is to ship this feature, I think @marco-ippolito's PR is a better layout of this.

Im happy if you want to takeover

[ShogunPanda]

LGTM!

[tniessen]

FWIW, there were two previous attempts at this: #51182 and #51370. I am not sure folks are convinced that we want to support this kind of arbitrary threshold.

[mcollina]

statements should be added too

[OnlyAviv]

statements should be added too

IIRC the coverage reporter doesn't support statements (but maybe it should in the future 🤔)

[MoLow]

there's always 4 thresholds: branch, function, line, and statements. can statements be added also?

@ljharb @mcollina statements coverage are only available when using instrumentation coverage from tools such as isnatbul.
V8 coverage dosent report that, you can see that c8 for instance doesn't have that either

[MoLow]

My main concern with previous attempts was lack of source map support. Now that we have that I thinks this is ok.
Also see my previous comment regarding lack of statements coverage (we might want to document that limitation, but it is a limitation that exists also in the coverage report/event, regardless of thresholds)

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/54429
✔  Done loading data for nodejs/node/pull/54429
----------------------------------- PR info ------------------------------------
Title      test_runner: add support for coverage thresholds (#54429)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     RedYetiDev:coverage-threshold -> nodejs:main
Labels     semver-minor, notable-change, cli, author ready, coverage, commit-queue-squash, test_runner
Commits    2
 - test_runner: add support for coverage thresholds
 - (skip if inspector disabled)
Committers 1
 - RedYetiDev <[email protected]>
PR-URL: https://github.com/nodejs/node/pull/54429
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Paolo Insogna <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Marco Ippolito <[email protected]>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/54429
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Paolo Insogna <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Marco Ippolito <[email protected]>
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last approving review:
   ⚠  - (skip if inspector disabled)
   ℹ  This PR was created on Sat, 17 Aug 2024 22:18:01 GMT
   ✔  Approvals: 5
   ✔  - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/54429#pullrequestreview-2247072628
   ✔  - Paolo Insogna (@ShogunPanda) (TSC): https://github.com/nodejs/node/pull/54429#pullrequestreview-2244734218
   ✔  - Moshe Atlow (@MoLow) (TSC): https://github.com/nodejs/node/pull/54429#pullrequestreview-2246968179
   ✔  - Colin Ihrig (@cjihrig): https://github.com/nodejs/node/pull/54429#pullrequestreview-2255436655
   ✔  - Marco Ippolito (@marco-ippolito) (TSC): https://github.com/nodejs/node/pull/54429#pullrequestreview-2255452813
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2024-08-23T13:36:18Z: https://ci.nodejs.org/job/node-test-pull-request/61378/
- Querying data for job/node-test-pull-request/61378/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/10529382777

[nodejs-github-bot]

Landed in 9edf4a0