Core + Addon Test: Refactor test API and fix total test count

[ghengeveld]

Closes #29541

What I did

• TESTING_MODULE_RUN_ALL_REQUEST is removed in favor of TESTING_MODULE_RUN_REQUEST. So we have a single event to run tests now.
• The payload to TESTING_MODULE_RUN_REQUEST is greatly simplified:

Before:

export type TestingModuleRunRequestPayload = {
  providerId: TestProviderId;
  payload: {
    stories: { id: string; name: string }[];
    importPath: string;
    componentPath: string;
  }[];
};

After:

export type TestingModuleRunRequestPayload = {
  providerId: TestProviderId;
  indexUrl: string; // e.g. http://localhost:6006/index.json
  storyIds?: string[]; // ['button--primary', 'button--secondary']
};

• The VitestManager now fetches the indexUrl to be able to match Vitest test specs (which only have a file path) to the storyIds provided. This way we make sure that only tests are executed for stories that exist in the index (which may otherwise not be true considering Vitest uses a different glob and globbing library than Storybook does).
• testNamePattern is now only set for single story runs, rather than whenever stories are specified in the request, because this testNamePattern trick will only reliably work when targeting a single test file, and we don't / won't have a UI pattern to run a subset of stories on a single component even though the API would support it. The story name is also retrieved from the index rather than passed along in the request payload.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	78.4 MB	78.4 MB	0 B	2.41	0%
initSize	144 MB	144 MB	-5.62 kB	-0.19	0%
diffSize	65.1 MB	65.1 MB	-5.62 kB	-0.33	0%
buildSize	6.83 MB	6.83 MB	-1.78 kB	-0.46	0%
buildSbAddonsSize	1.51 MB	1.51 MB	-211 B	-Infinity	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.86 MB	1.86 MB	-1.42 kB	-0.45	-0.1%
buildSbPreviewSize	271 kB	271 kB	-147 B	-Infinity	-0.1%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.83 MB	3.83 MB	-1.78 kB	-0.46	0%
buildPreviewSize	3 MB	3 MB	0 B	0.65	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	6.7s	23.2s	16.4s	0.89	70.8%
generateTime	19.8s	19.3s	-546ms	-0.72	-2.8%
initTime	13.8s	13.8s	9ms	-0.38	0.1%
buildTime	9.4s	8.1s	-1s -322ms	-0.57	-16.2%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	5.7s	5.7s	-60ms	-0.19	-1.1%
devManagerResponsive	3.4s	3.3s	-125ms	-0.79	-3.7%
devManagerHeaderVisible	551ms	516ms	-35ms	-0.78	-6.8%
devManagerIndexVisible	629ms	547ms	-82ms	-1.27	🔰-15%
devStoryVisibleUncached	921ms	657ms	-264ms	-1.59	🔰-40.2%
devStoryVisible	626ms	546ms	-80ms	-1.06	-14.7%
devAutodocsVisible	545ms	484ms	-61ms	-0.3	-12.6%
devMDXVisible	532ms	475ms	-57ms	-0.57	-12%
buildManagerHeaderVisible	539ms	527ms	-12ms	-0.43	-2.3%
buildManagerIndexVisible	554ms	536ms	-18ms	-0.49	-3.4%
buildStoryVisible	541ms	528ms	-13ms	-0.39	-2.5%
buildAutodocsVisible	451ms	449ms	-2ms	-0.36	-0.4%
buildMDXVisible	447ms	450ms	3ms	-0.32	0.7%

Greptile Summary

This PR refactors Storybook's test API to simplify test execution and improve test progress reporting accuracy.

• Consolidates test execution into single TESTING_MODULE_RUN_REQUEST event, removing separate "run all" event
• Simplifies test request payload to use indexUrl and optional storyIds instead of detailed story information
• Improves test progress accuracy by using static story count from index rather than Vitest's dynamic test count
• Adds server-side story index fetching to ensure tests only run for existing stories
• Implements new context menu system for test providers with improved status handling and accessibility

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 23aa9b8. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 1 target

• nx run-many -t build --parallel=3

---

Sent with 💌 from NxCloud.

[storybook-pr-benchmarking]

Package Benchmarks

^{Commit: 23aa9b8, ran on 25 November 2024 at 10:11:41 UTC}

No significant changes detected, all good. 👏

[valentinpalkovic]

The Vite Backend part LGTM.

[greptile-apps]

_{41 file(s) reviewed, 25 comment(s)
Edit PR Review Bot Settings | Greptile}

[JReinhold]

Based on the diff I see, I think you need to pull in next

[JReinhold]

Apart from the failing skip test, I think it looks good.

[ndelangen]

@ghengeveld can we get this in a green state please, it's my base-branch for this work:
#29662