Project Information GitHub Actions

The relevant code for GitHub Actions within TFB is within:

• The .github/workflows directory, which contains the code that automates the verification of each framework
• The toolset directory, which contains the utility code that TFB uses to automate installing, launching, load testing, and terminating each framework

GitHub Actions Usage

This section details how TFB uses GitHub Actions. At a high level, there is a GitHub Actions workflow defined that triggers a build every time new commits are pushed to a branch or to a pull request. Each push causes GitHub Actions to launch a virtual machine, checkout the code, run an installation, and run a verification.

GitHub Actions Terminology

Each push to GitHub triggers a new GitHub Actions workflow. Each workflow contains a number of semi-independent jobs. Each job is run on an isolated virtual machine called a runner, and contains a series of steps. Each step may contain either commands to run in the shell, or an action. An action is a configurable script published to the GitHub Actions marketplace or provided by GitHub itself.

Our project has a setup job which then creates one job for each language directory with tests to run, e.g. one job for Go, one job for Java, etc. Each job gets its own runner virtual machine that runs the installation for each framework under that language directory and verifies the framework's output using --mode verify.

The .github/workflows/build.yml file initially generates the build matrix, which is the set of jobs that should be run for each workflow. Our build matrix lists each language directory with tests to run, which causes each workflow to have one job for each of those directories.

If the workflow is canceled, then the workflow is marked canceled and all remaining jobs are stopped/canceled. If any job returns failed, then the workflow is marked failed. The workflow is marked passing only if every job returns passing.

GitHub Actions' Limits

GitHub Actions is a free service for public repositories (free/paid tiers available for private repositories), and for public repositories imposes minimal limits.

Each time someone pushes new commits to a branch (or to a pull request), a new workflow is triggered that contains up to ~40 jobs, one for each language directory with tests to run. This can be resource intensive for certain pull requests, so it is important to understand GitHub Actions' limits.

Hours per Workflow: 72 hours maximum. Not really an issue at all, we usually end up at around 3.5 hours for a full build.

Minutes Per Job: 360 minutes maximum. None of the jobs we run hit this limit (most take 10-15 minutes total, some take up to 3 hours).

Max Concurrent Jobs: Typically around 5-10 jobs (for public repositories it's dependent on available runners from GitHub). Discussed below.

Dealing with GitHub Actions' Limits

Max Concurrent Jobs: If needed, the TechEmpower team can manually cancel workflows (but not individual jobs) directly from the Github Actions interface.

Tricks and Tips for GitHub Actions

Use your own fork's GitHub Actions: We cannot stress this enough. Your own Actions will help you work 10x faster, and it's easy to do (and free). Just push a commit, and a workflow will start up to test any changes from that commit.

How to see logs in GitHub Actions: You may occasionally need to see the logs. Fortunately these are accessible through the GitHub Actions UI directly for reading within each step, as well as downloadable per-job once each job completes. Until the job is complete, the logs may be viewed in real-time.

Advanced GitHub Actions Details

Partial vs Full Verification

GitHub Actions first checks if there is any reason to run a verification for this framework. This check uses git diff to list the files that have been modified. If files relevant to this framework have not been modified, then GitHub Actions doesn't bother running the installation (or the verification) as nothing will have changed from the last workflow. We call this a partial verification, and if only one framework has been modified then the entire workflow will complete within 10-15 minutes.

If any of the setup scripts within the toolset/ has changed, github_actions_diff attempts to find only the frameworks that would be affected by this and run only those jobs. Other files within the toolset/ that are not setup scripts will run all jobs. We call this a full verification, and the entire workflow will complete within 3-5 hours.

GitHub Actions has been set up to not queue up a job for languages that we do not need to test. If any language is omitted for this reason, that is a partial verification. The final status for the verification of the pull request will therefore depend on the exit status of the jobs that are run - if they return passed then the entire workflow will be marked passed, and similarly for failed.

For example, if files inside aspnetcore/ are modified as part of a pull request, then every job but CSharp (the language directory under which aspnetcore resides) will be omitted, and only CSharp will be tested. The return code of the CSharp job will then determine the final exit status of the workflow.

GitHub Actions Message Hooks

Our GitHub Actions workflow provides a few commit message hooks to help make dealing with GitHub Actions as well as the Partial vs Full Verification a little easier. You can add any of the following to your commit message when pushing to a GitHub Actions enabled repo. Each GitHub Actions workflow evaluates the most recent commit message.

• [ci skip] - If this is found in your commit message, a workflow will still trigger, but no language jobs will be created, only the setup job.
• [ci run-all] - This will run all the jobs despite what files have been changed.
• [ci fw <testdirs>] - This will run the tests for the selected framework directories in addition to the jobs that would normally trigger from the files that have been changed. Example: [ci fw JavaScript/express Java/gemini]
• [ci fw-only <testdirs>] - The same as above except it will only run these tests despite the files that have been modified. Example: [ci fw-only JavaScript/express Java/gemini]
• [ci lang <testlang>] - This will run the tests for the selected language in addition to the jobs that would normally trigger from the files that have been changed. Example: [ci lang JavaScript Java]
• [ci lang-only <testlang>] - The same as above except it will only run these language tests despite the files that have been modified. Example: [ci fw-only JavaScript Java]

Running GitHub Actions in a Fork

No setup is needed to enable GitHub Actions in your fork, and it is highly valuable as you have personal limits on runners and can therefore see results from GitHub Actions much more quickly than you could when the GitHub Actions queue for the base repository is full of workflows queue awaiting verification.

Important Notes About GitHub Actions

If you make changes to configuration files, or files outside the frameworks directory, GitHub Actions will run tests on all existing frameworks. For this reason, it may appear that your tests have failed. Be sure to check your GitHub Actions workflow by clicking on the checkmark or red 'X' to dig into your specific test/language.

Be sure to keep your Pull Request branch up-to-date with the target branch, otherwise our diffing tool may detect additional changes causing unwanted tests to run.