Test package for ecosystem libraries

[Fryuni]

Body

Summary

This proposal suggests a new @astrojs/test-utils package (name TBD) containing first-party tools for library, adapter and integration authors to test their packages reliably. Such a package would enable authors to maintain their own projects without breaking package isolation nor having to replicate parts of Astro implementation on their own test suite.

Background & Motivation

There are docs around testing an Astro site but, currently, Astro lacks official tooling for testing integrations and libraries. There has been questions regarding this on GH and on Discord, for example:

• Integration with Vitest #74
• https://discord.com/channels/830184174198718474/1178906267989843998/1178906267989843998
• https://discord.com/channels/830184174198718474/1167265554101436488/1167265554101436488

Sometimes we point the users to the adapters repo as a source for them to copy into their own projects, but even within the adapters repo there has been problems.

This lack of test tooling also affects official Astro projects outside of the astro package. Issues and discussions about providing a test package having beeing going on for a while.

Goals

• Provide an official testing utility specifically designed for Astro integrations, libraries, and components.
• Simplify the process of creating, running, and maintaining tests for projects on the Astro ecosystem.
• Not force any specific testing library (should be usable with vitest, Node tests, Playwright, Cypress, etc.)

Examples

Astro Core tests

astro has a test-utils module that defines a great set of utilities for testing.

A fixture is loaded and a collection of method to work with that fixture are provided. The API usability follows this style:

const fixture = await loadFixture({
	root: './fixtures/astro-basic/',
});
await fixture.build();
const previewServer = await fixture.preview();

This is a good starting point, but has some problems:

• It is focused on testing Astro internals, not libraries made to be used with Astro.
• It relies heavily on Astro internals, which are not exposed for other packages to use.
• Astro's core tests are in JS, so there was no need to comply with TS for this test utils module. For arbitrary projects in the ecosystem, the fixture should be typed.

Third party implementation

I ported the utilities from core into an external package to use for my own tests, @inox-tools/astro-tests.

Here is an example of it being used.

It mostly resolves the problem with the module from astro's tests, but it still has some disadvantages that can be resolved from Astro core:

• The function mergeConfig that combines inline configuration with configuration loaded from an Astro config file is not exported. This function is similar to a function of the same name from Vite, but Vite's function is exported. Ideally this function could be exported from Astro so it would not have to be copied into the package.
• Similarly, the function validateConfig that fills default values and apply transformations on the configuration is also not exported. This is required to resolve a value from the configuration that is not directly set on the test but used for the test tolling, like the outDir and srcDir.
• The exported build function hides the option to re-use the instantiated WASM module from the compiler. This causes a performance degradation on tests building projects.

[matthewp]

Whether we create a new package or not, we still need to expose these internals. Can you provided the suggested signatures and exports of what you'd like to see exposed? For example, mergeConfig is currently typed as mergeConfig(defaults: Record<string, any>, overrides: Record<string, any>): Record<string, any>), is that ok? Also, where should this be exposed from? astro/config? Same question for the other internal functions.

[Fryuni]

I think both mergeConfig and validateConfig would fit nicely being exported from astro/config.

The signature of mergeConfig can be changed to this:

function mergeConfig<C extends AstroConfig | AstroInlineConfig>(
	defaults: C,
	overrides: DeepPartial<C>,
	isRoot = true,
): C

I didn't find any use in the codebase where that isRoot is passed, so it is effectively always true. We could remove it.

validateConfig already has the proper signature IMO, it accepts anything as the input and then validates and transforms the input, returning an AstroConfig:

function validateConfig(
	userConfig: any,
	root: string,
	cmd: string,
): Promise<AstroConfig>

Lastly, I don't think this wrapper to hide the second parameter is needed. Neither this internal modifier.

[matthewp]

Why AstroConfig | AstroInlineConfig? Is this running the schema to convert it?

[Fryuni]

It is used to merge values both before and after resolving.

• It merges AstroInlineConfig when combining the config file with what is passed inline to the build/dev command. The result is used for resolving.
• It merges AstroConfig when calling updateConfig from an integration.