tools: add a daily wpt.fyi synchronized report upload #46498

panva · 2023-02-04T14:22:09Z

This PR introduces a daily job for uploading upstream synchronized WPT executed in four versions of Node.js to the wpt.fyi API.

Test run generated reports: https://github.com/nodejs/node-auto-test/suites/10778564147/artifacts/541679841

Example: Deno has their results submitted next to browsers as well

This PR has two commits, the first one that updates the WPTRunner and adds make target to generate the reports which we'll backport to v19.x, v18.x, and maybe v16.x.

The four versions are:

3 based on actions/setup-node - current, lts/* (always Active LTS), lts/-1 (most often the Maintenance LTS)
latest nightly build per https://nodejs.org/download/nightly/index.json

The job is setup to always execute the respective version's WPTRunner and the supported WPT subset but with the latest synchronized WPT fixtures.

The second commit introduces the daily job's workflow yaml file.

I have requested the credentials for accessing the API. This is a draft until we get them and I retest the flow via nodejs/node-auto-test again once we get the credentials and set them as either organization or repository secrets.

v19.x lands clean
v18.x I will open backport PR for the WPTRunner and make target
v16.x I will open backport PR for the WPTRunner and make target

The workflow file gracefully handles cases when the backport was not done yet and the way the Node.js versions are determined requires no upkeep.

Closes #46443

nodejs-github-bot · 2023-02-04T14:22:13Z

Review requested:

@nodejs/actions

panva · 2023-02-04T14:23:35Z

cc @anonrig

test/common/wpt.js

Makefile

test/common/wpt.js

richardlau · 2023-02-04T14:50:46Z

* the node version built from the current `main`

If the job is only running daily, you could save the build cost by using a nightly build from https://nodejs.org/download/nightly/. actions/setup-node can use nightlies: https://github.com/actions/setup-node/blob/main/docs/advanced-usage.md#nightly-versions

panva · 2023-02-04T14:54:04Z

If the job is only running daily, you could save the build cost by using a nightly build from nodejs.org/download/nightly. actions/setup-node can use nightlies: actions/setup-node@main/docs/advanced-usage.md#nightly-versions

Great suggestion! Is just nightly going to be a valid target though? The examples suggest It needs a major.x-nightly target which would require upkeep.

panva · 2023-02-04T18:46:00Z

Run actions/setup-node@v3
  with:
    node-version: nightly
Error: Unable to find Node version 'nightly' for platform linux and architecture x64.

anonrig

Great work @panva!

.github/workflows/daily-wpt-fyi.yml

test/common/wpt.js

test/wpt/test-webcrypto.js

panva · 2023-02-05T10:10:14Z

@richardlau with a little bit of convincing but it's now using nightly instead of building from main.

panva · 2023-02-05T10:45:18Z

@KhafraDev It would be amazing if we started thinking about executing fetch()-related WPTs in nodejs/node as well so that they would also get submitted. Just something to think about.

KhafraDev · 2023-02-05T17:54:13Z

I don't think it's a good idea to run fetch WPTs in node - node doesn't implement fetch and the WPT runner isn't powerful enough to run the tests. Would it be possible for undici to run the WPTs, generate a report, and then for node to fetch the report and combine the results with other tests?

panva · 2023-02-05T17:59:02Z

I don't think it's a good idea to run fetch WPTs in node - node doesn't implement fetch and the WPT runner isn't powerful enough to run the tests. Would it be possible for undici to run the WPTs, generate a report, and then for node to fetch the report and combine the results with other tests?

Let me think more on this.

Could you port your improvements to the runner back to node?

KhafraDev · 2023-02-05T18:41:19Z

I'm not sure tbh. The biggest difference is that undici's runner needs to create multiple servers (1 for fetch tests and 1 for websocket tests). With that, it also runs in its own process to ensure that if a tests, for example, blocks the thread or mutates a prototype, there won't be any side effects to the server.

There's also some slight differences between the two - undici will respect META attributes (timeout, etc.) while node's seems to only care about scripts to inject. Undici's will open x amount of workers in parallel while node's opens 1 at a time, etc., etc..

But then, a majority of tests use relative path URLs, and undici's fetch can't do anything with these urls by default (since node doesn't have a default origin and file:// urls can't be fetched). Undici exposes setGlobalOrigin to set a... global origin... that's then used as a base to parse relative urls. I'm unsure how node would do this since undici doesn't export it in its fetch bundle.

All of those issues could be fixed, but I don't really see the benefit if I'm being honest.

panva · 2023-02-05T18:56:57Z

We should then look into undici producing wpt reports that the job here could pull and submit as one run chunked into 2.

panva · 2023-02-17T15:00:26Z

cc @nodejs/actions this should be ready to land (#46498 (comment))

nodejs-github-bot · 2023-02-17T15:59:38Z

Landed in eb2a1ab...65854f1

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: nodejs#46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]> Backport-PR-URL: https://github.com/nodejs/node/pull/TODO

PR-URL: nodejs#46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]> Backport-PR-URL: nodejs#46767

PR-URL: nodejs#46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]> Backport-PR-URL: nodejs#46768

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]> Backport-PR-URL: #46767

PR-URL: #46498 Backport-PR-URL: #46768 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [node](https://github.com/nodejs/node) | stage | minor | `16.19.1-bullseye` -> `16.20.0-bullseye` | --- ### Release Notes <details> <summary>nodejs/node</summary> ### [`v16.20.0`](https://github.com/nodejs/node/releases/tag/v16.20.0): 2023-03-29, Version 16.20.0 'Gallium' (LTS), @BethGriggs [Compare Source](nodejs/node@v16.19.1...v16.20.0) ##### Notable Changes - **deps:** - update undici to 5.20.0 (Node.js GitHub Bot) [#46711](nodejs/node#46711) - update c-ares to 1.19.0 (Michaël Zasso) [#46415](nodejs/node#46415) - upgrade npm to 8.19.4 (npm team) [#46677](nodejs/node#46677) - update corepack to 0.17.0 (Node.js GitHub Bot) [#46842](nodejs/node#46842) - **(SEMVER-MINOR)** **src**: add support for externally shared js builtins (Michael Dawson) [#44376](nodejs/node#44376) ##### Commits - \[[`de6dd67790`](nodejs/node@de6dd67790)] - **crypto**: avoid hang when no algorithm available (Richard Lau) [#46237](nodejs/node#46237) - \[[`4617512788`](nodejs/node@4617512788)] - **crypto**: ensure auth tag set for chacha20-poly1305 (Ben Noordhuis) [#46185](nodejs/node#46185) - \[[`24972164fc`](nodejs/node@24972164fc)] - **deps**: update undici to 5.20.0 (Node.js GitHub Bot) [#46711](nodejs/node#46711) - \[[`85f88c6a8d`](nodejs/node@85f88c6a8d)] - **deps**: V8: cherry-pick [`90be99f`](nodejs/node@90be99fab31c) (Michaël Zasso) [#46646](nodejs/node#46646) - \[[`b4ebe6d47b`](nodejs/node@b4ebe6d47b)] - **deps**: update c-ares to 1.19.0 (Michaël Zasso) [#46415](nodejs/node#46415) - \[[`56cbc7fdda`](nodejs/node@56cbc7fdda)] - **deps**: V8: cherry-pick [`c2792e5`](nodejs/node@c2792e58035f) (Jiawen Geng) [#44961](nodejs/node#44961) - \[[`7af9bdb31e`](nodejs/node@7af9bdb31e)] - **deps**: upgrade npm to 8.19.4 (npm team) [#46677](nodejs/node#46677) - \[[`962a7471b5`](nodejs/node@962a7471b5)] - **deps**: update corepack to 0.17.0 (Node.js GitHub Bot) [#46842](nodejs/node#46842) - \[[`748bc96e35`](nodejs/node@748bc96e35)] - **deps**: update corepack to 0.16.0 (Node.js GitHub Bot) [#46710](nodejs/node#46710) - \[[`a467782499`](nodejs/node@a467782499)] - **deps**: update corepack to 0.15.3 (Node.js GitHub Bot) [#46037](nodejs/node#46037) - \[[`1913b6763d`](nodejs/node@1913b6763d)] - **deps**: update corepack to 0.15.2 (Node.js GitHub Bot) [#45635](nodejs/node#45635) - \[[`809371a15f`](nodejs/node@809371a15f)] - **module**: require.resolve.paths returns null with node schema (MURAKAMI Masahiko) [#45147](nodejs/node#45147) - \[[`086bb2f8d4`](nodejs/node@086bb2f8d4)] - ***Revert*** "**src**: let http2 streams end after session close" (Rich Trott) [#46721](nodejs/node#46721) - \[[`6a01d39120`](nodejs/node@6a01d39120)] - **(SEMVER-MINOR)** **src**: add support for externally shared js builtins (Michael Dawson) [#44376](nodejs/node#44376) - \[[`d081032a60`](nodejs/node@d081032a60)] - **test**: fix test-net-connect-reset-until-connected (Vita Batrla) [#46781](nodejs/node#46781) - \[[`efe1be47ec`](nodejs/node@efe1be47ec)] - **test**: skip test depending on `overlapped-checker` when not available (Antoine du Hamel) [#45015](nodejs/node#45015) - \[[`fc47d58abe`](nodejs/node@fc47d58abe)] - **test**: remove cjs loader from stack traces (Geoffrey Booth) [#44197](nodejs/node#44197) - \[[`cf76d0790d`](nodejs/node@cf76d0790d)] - **test**: fix WPT title when no META title is present (Filip Skokan) [#46804](nodejs/node#46804) - \[[`0d1485b924`](nodejs/node@0d1485b924)] - **test**: fix default WPT titles (Filip Skokan) [#46778](nodejs/node#46778) - \[[`088e9cde3d`](nodejs/node@088e9cde3d)] - **test**: add WPTRunner support for variants and generating WPT reports (Filip Skokan) [#46498](nodejs/node#46498) - \[[`908c4dff44`](nodejs/node@908c4dff44)] - **test**: mark test-crypto-key-objects flaky on Linux (Richard Lau) [#46684](nodejs/node#46684) - \[[`768e56227e`](nodejs/node@768e56227e)] - **tools**: make `utils.SearchFiles` deterministic (Bruno Pitrus) [#44496](nodejs/node#44496) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/mwalbeck/docker-cyberchef/pulls/187 Co-authored-by: renovate-bot <[email protected]> Co-committed-by: renovate-bot <[email protected]>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [node](https://github.com/nodejs/node) | stage | minor | `16.19.1-bullseye-slim` -> `16.20.0-bullseye-slim` | --- ### Release Notes <details> <summary>nodejs/node</summary> ### [`v16.20.0`](https://github.com/nodejs/node/releases/tag/v16.20.0): 2023-03-29, Version 16.20.0 'Gallium' (LTS), @BethGriggs [Compare Source](nodejs/node@v16.19.1...v16.20.0) ##### Notable Changes - **deps:** - update undici to 5.20.0 (Node.js GitHub Bot) [#46711](nodejs/node#46711) - update c-ares to 1.19.0 (Michaël Zasso) [#46415](nodejs/node#46415) - upgrade npm to 8.19.4 (npm team) [#46677](nodejs/node#46677) - update corepack to 0.17.0 (Node.js GitHub Bot) [#46842](nodejs/node#46842) - **(SEMVER-MINOR)** **src**: add support for externally shared js builtins (Michael Dawson) [#44376](nodejs/node#44376) ##### Commits - \[[`de6dd67790`](nodejs/node@de6dd67790)] - **crypto**: avoid hang when no algorithm available (Richard Lau) [#46237](nodejs/node#46237) - \[[`4617512788`](nodejs/node@4617512788)] - **crypto**: ensure auth tag set for chacha20-poly1305 (Ben Noordhuis) [#46185](nodejs/node#46185) - \[[`24972164fc`](nodejs/node@24972164fc)] - **deps**: update undici to 5.20.0 (Node.js GitHub Bot) [#46711](nodejs/node#46711) - \[[`85f88c6a8d`](nodejs/node@85f88c6a8d)] - **deps**: V8: cherry-pick [`90be99f`](nodejs/node@90be99fab31c) (Michaël Zasso) [#46646](nodejs/node#46646) - \[[`b4ebe6d47b`](nodejs/node@b4ebe6d47b)] - **deps**: update c-ares to 1.19.0 (Michaël Zasso) [#46415](nodejs/node#46415) - \[[`56cbc7fdda`](nodejs/node@56cbc7fdda)] - **deps**: V8: cherry-pick [`c2792e5`](nodejs/node@c2792e58035f) (Jiawen Geng) [#44961](nodejs/node#44961) - \[[`7af9bdb31e`](nodejs/node@7af9bdb31e)] - **deps**: upgrade npm to 8.19.4 (npm team) [#46677](nodejs/node#46677) - \[[`962a7471b5`](nodejs/node@962a7471b5)] - **deps**: update corepack to 0.17.0 (Node.js GitHub Bot) [#46842](nodejs/node#46842) - \[[`748bc96e35`](nodejs/node@748bc96e35)] - **deps**: update corepack to 0.16.0 (Node.js GitHub Bot) [#46710](nodejs/node#46710) - \[[`a467782499`](nodejs/node@a467782499)] - **deps**: update corepack to 0.15.3 (Node.js GitHub Bot) [#46037](nodejs/node#46037) - \[[`1913b6763d`](nodejs/node@1913b6763d)] - **deps**: update corepack to 0.15.2 (Node.js GitHub Bot) [#45635](nodejs/node#45635) - \[[`809371a15f`](nodejs/node@809371a15f)] - **module**: require.resolve.paths returns null with node schema (MURAKAMI Masahiko) [#45147](nodejs/node#45147) - \[[`086bb2f8d4`](nodejs/node@086bb2f8d4)] - ***Revert*** "**src**: let http2 streams end after session close" (Rich Trott) [#46721](nodejs/node#46721) - \[[`6a01d39120`](nodejs/node@6a01d39120)] - **(SEMVER-MINOR)** **src**: add support for externally shared js builtins (Michael Dawson) [#44376](nodejs/node#44376) - \[[`d081032a60`](nodejs/node@d081032a60)] - **test**: fix test-net-connect-reset-until-connected (Vita Batrla) [#46781](nodejs/node#46781) - \[[`efe1be47ec`](nodejs/node@efe1be47ec)] - **test**: skip test depending on `overlapped-checker` when not available (Antoine du Hamel) [#45015](nodejs/node#45015) - \[[`fc47d58abe`](nodejs/node@fc47d58abe)] - **test**: remove cjs loader from stack traces (Geoffrey Booth) [#44197](nodejs/node#44197) - \[[`cf76d0790d`](nodejs/node@cf76d0790d)] - **test**: fix WPT title when no META title is present (Filip Skokan) [#46804](nodejs/node#46804) - \[[`0d1485b924`](nodejs/node@0d1485b924)] - **test**: fix default WPT titles (Filip Skokan) [#46778](nodejs/node#46778) - \[[`088e9cde3d`](nodejs/node@088e9cde3d)] - **test**: add WPTRunner support for variants and generating WPT reports (Filip Skokan) [#46498](nodejs/node#46498) - \[[`908c4dff44`](nodejs/node@908c4dff44)] - **test**: mark test-crypto-key-objects flaky on Linux (Richard Lau) [#46684](nodejs/node#46684) - \[[`768e56227e`](nodejs/node@768e56227e)] - **tools**: make `utils.SearchFiles` deterministic (Bruno Pitrus) [#44496](nodejs/node#44496) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/mwalbeck/docker-jellyfin-livestream/pulls/243 Co-authored-by: renovate-bot <[email protected]> Co-committed-by: renovate-bot <[email protected]>

nodejs-github-bot added build Issues and PRs related to build files or the CI. meta Issues and PRs related to the general management of the project. needs-ci PRs that need a full CI run. labels Feb 4, 2023

panva commented Feb 4, 2023

View reviewed changes

test/common/wpt.js Show resolved Hide resolved

panva commented Feb 4, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

panva commented Feb 4, 2023

View reviewed changes

test/common/wpt.js Outdated Show resolved Hide resolved

panva force-pushed the wptfyi branch from 42c557b to 37dd5bf Compare February 4, 2023 14:48

anonrig reviewed Feb 4, 2023

View reviewed changes

panva force-pushed the wptfyi branch 4 times, most recently from 20db2fc to cc03fea Compare February 4, 2023 21:06

panva added the commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. label Feb 4, 2023

panva force-pushed the wptfyi branch from cc03fea to c7a5180 Compare February 5, 2023 10:07

panva force-pushed the wptfyi branch from c7a5180 to 7b2e5ea Compare February 5, 2023 10:32

panva mentioned this pull request Feb 5, 2023

Node.js test runner registration request web-platform-tests/wpt.fyi#3144

Closed

panva force-pushed the wptfyi branch 3 times, most recently from 1be365b to 0bc7bd4 Compare February 5, 2023 21:27

richardlau approved these changes Feb 17, 2023

View reviewed changes

panva added the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 17, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 17, 2023

nodejs-github-bot closed this Feb 17, 2023

nodejs-github-bot pushed a commit that referenced this pull request Feb 17, 2023

test: add WPTRunner support for variants and generating WPT reports

ffdbdf4

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

nodejs-github-bot pushed a commit that referenced this pull request Feb 17, 2023

tools: add a daily wpt.fyi synchronized report upload

65854f1

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

panva deleted the wptfyi branch February 17, 2023 18:02

github-actions bot mentioned this pull request Feb 18, 2023

CI Reliability 2023-02-18 nodejs/reliability#499

Open

32 tasks

MylesBorins pushed a commit that referenced this pull request Feb 18, 2023

test: add WPTRunner support for variants and generating WPT reports

db266df

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

MylesBorins pushed a commit that referenced this pull request Feb 18, 2023

tools: add a daily wpt.fyi synchronized report upload

55d7288

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

MylesBorins mentioned this pull request Feb 19, 2023

v19.7.0 proposal #46725

Merged

MylesBorins pushed a commit that referenced this pull request Feb 20, 2023

test: add WPTRunner support for variants and generating WPT reports

e60d3f2

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

MylesBorins pushed a commit that referenced this pull request Feb 20, 2023

tools: add a daily wpt.fyi synchronized report upload

f62b58a

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

KhafraDev mentioned this pull request Feb 21, 2023

test(wpt): add results to an existing WPT Report nodejs/undici#1944

Merged

This was referenced Feb 22, 2023

[v18.x backport] test: add WPTRunner support for variants and generating WPT reports #46767

Closed

[v16.x backport] test: add WPTRunner support for variants and generating WPT reports #46768

Closed

panva added backport-open-v16.x backport-open-v18.x Indicate that the PR has an open backport. labels Feb 22, 2023

juanarbol pushed a commit that referenced this pull request Mar 5, 2023

test: add WPTRunner support for variants and generating WPT reports

5f422c4

PR-URL: #46498 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]> Backport-PR-URL: #46767

juanarbol mentioned this pull request Mar 5, 2023

V18.15.0 proposal #46920

Merged

BethGriggs pushed a commit that referenced this pull request Mar 23, 2023

test: add WPTRunner support for variants and generating WPT reports

088e9cd

PR-URL: #46498 Backport-PR-URL: #46768 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Richard Lau <[email protected]>

BethGriggs mentioned this pull request Mar 27, 2023

v16.20.0 proposal #47272

Merged

panva mentioned this pull request Jan 16, 2024

Avoid structuredClone in Response() nodejs/undici#2617

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: add a daily wpt.fyi synchronized report upload #46498

tools: add a daily wpt.fyi synchronized report upload #46498

panva commented Feb 4, 2023 •

edited

Loading

nodejs-github-bot commented Feb 4, 2023

panva commented Feb 4, 2023

richardlau commented Feb 4, 2023

panva commented Feb 4, 2023 •

edited

Loading

panva commented Feb 4, 2023

anonrig left a comment

panva commented Feb 5, 2023

panva commented Feb 5, 2023

KhafraDev commented Feb 5, 2023

panva commented Feb 5, 2023

KhafraDev commented Feb 5, 2023

panva commented Feb 5, 2023

panva commented Feb 17, 2023 •

edited

Loading

nodejs-github-bot commented Feb 17, 2023

tools: add a daily wpt.fyi synchronized report upload #46498

tools: add a daily wpt.fyi synchronized report upload #46498

Conversation

panva commented Feb 4, 2023 • edited Loading

nodejs-github-bot commented Feb 4, 2023

panva commented Feb 4, 2023

richardlau commented Feb 4, 2023

panva commented Feb 4, 2023 • edited Loading

panva commented Feb 4, 2023

anonrig left a comment

Choose a reason for hiding this comment

panva commented Feb 5, 2023

panva commented Feb 5, 2023

KhafraDev commented Feb 5, 2023

panva commented Feb 5, 2023

KhafraDev commented Feb 5, 2023

panva commented Feb 5, 2023

panva commented Feb 17, 2023 • edited Loading

nodejs-github-bot commented Feb 17, 2023

panva commented Feb 4, 2023 •

edited

Loading

panva commented Feb 4, 2023 •

edited

Loading

panva commented Feb 17, 2023 •

edited

Loading