Feature/e2e tests workflow

[carstingaxion]

This PR allows to run automated and manual end-to-end tests, while sharing the same and existing, wp-env based, setup.

NEEDED, before merge

• Update all blocks to API Version 3, to be able to use editor.canvas easily.
  (Feature/e2e tests workflow #776 (comment))

Automated tests

Check the results of the e2e-tests action workflow at https://github.com/GatherPress/gatherpress/actions/workflows/e2e-tests.yml.

Manual testing

0. Have node installed
1. Have docker running
2. Open the plugin folder in a terminal
3. Choose one of the following options

   • Run Playwright in the background using

     npm run test:e2e
     

   • Run Playwright visually (and change what's happening) using

     npm run test:e2e:debug
     

     

   • Run Tests independently AND visually using the Playwright VSCode extension

Description of the Change

Closes #

How to test the Change

Changelog Entry

Changed existing end-to-tests to rely on wordpress/e2e-test-utils-playwright & setup GitHub actions workflow to run tests using the existing wp-env environment

Credits

Props @carstingaxion , @swissspidy

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added tests to cover my change.
• All new and existing tests pass.

[what-the-diff]

PR Summary

• Updated GitHub Testing Triggers and Concurrency
  The end-to-end (e2e) testing setup saw additions of new triggers and concurrency settings, improving efficiency of test scenarios.

• Refining Ignored Files For Cleaner Repository
  The artifacts/ directory, likely to contain temporary or generated files, has been added to the .gitignore file. This ensures a cleaner repository by avoiding unneeded files from being tracked.

• Configuration Adjustments For Testing Environment
  Certain changes were made in the .wp-env.json file which adjusted settings related to testing environment. Some mappings along with port and environment variables were removed.

• Introduction of Developer Documentation
  A README.md file has been added to the docs/developer/e2e-tests/ directory containing insightful details about end to end testing.

• Guide For Test Execution and Additional Dependencies
  Script updates and additional dependencies were added to package.json for better handling of e2e tests.

• Streamlined Testing Framework Configurations
  The playwright.config.js file has been removed and replaced with a playwright.config.ts file. This introduces TypeScript based configurations enhancing readability and maintainability.

• Preparation Scripts For Test Execution
  A global-setup.ts file has been added which will contain scripts to set up the test execution, ensuring conditions are optimal before tests are begun.

• Additional Test Utility Files
  A new directory test/e2e/test-utils/ was created with utility files index.ts and test.ts that will facilitate writing and running tests more effectively.

• Tests File Renaming and New Tests
  Files in test/e2e/tests/ directory have been renamed for consistency and readability. New test files gatherpress-blocks-editor.spec.ts have been added to enhance the test coverage.

• TypeScript Configuration For Project
  A tsconfig.json for TypeScript has been added to guide the TypeScript compiler on how to generate JavaScript files from TypeScript source.

[carstingaxion]

Hello @swissspidy , nice that you take the time & have a look; Thank you!

I've removed the globalSetup & the env variables, like you said. I'm absolutely fine removing as lot, as possible from not needed overhead!

While there are many tests failing now, because of my changes; let's ignore most of them for now. Those are the existing tests, that I haven't touched yet.

But let us please focus on gatherpress-blocks-editor.spec.ts which has two tests only and should help to illustrate the new things related to the e2e-test-utils-playwright package.

EDIT: Please take a look at https://github.com/GatherPress/gatherpress/actions/runs/10348428519/job/28640652754?pr=776#step:8:231 which is the result of one of the simplest paragraph tests from gutenberg, which fails.

[swissspidy]

A screenshot of the browser is more helpful than one from the terminal, but thanks :)

The test you copied from Gutenberg does not work in your use case because your editor is not iframed.

The post editor will be iframed if all registered blocks have a Block API version 3 or higher and the editor has no classic meta boxes below the blocks.

This is the case in the Gutenberg test environment, but not in yours, because all the GatherPress blocks are using apiVersion 2.

So if you are interacting with editor.canvas, you need to adjust your locator usage accordingly because editor.canvasisn't an iframe, so.locator( ':root' )` won't find anything. Alternatively of course bump your apiVersion to ensure the editor is iframed.

[carstingaxion]

A screenshot of the browser is more helpful than one from the terminal, but thanks :)

I know, sorry. But I didn't knew, what kind of screenshot would make sense.

The test you copied from Gutenberg does not work in your use case because your editor is not iframed.

The post editor will be iframed if all registered blocks have a Block API version 3 or higher and the editor has no classic meta boxes below the blocks.

This is the case in the Gutenberg test environment, but not in yours, because all the GatherPress blocks are using apiVersion 2.

So if you are interacting with editor.canvas, you need to adjust your locator usage accordingly because editor.canvasisn't an iframe, so.locator( ':root' )` won't find anything. Alternatively of course bump your apiVersion to ensure the editor is iframed.

This was (I guess) the holy-grail answer, I was looking for, for days now! Thank you, very much @swissspidy !!!