Use Turborepo for tests and add turbo cache for CI eslint and unit-tests #3458

menghif · 2022-04-11T14:54:58Z

Issue This PR Addresses

Type of Change

Bugfix: Change which fixes an issue
New Feature: Change which adds functionality
Documentation Update: Change which improves documentation
UI: Change which improves UI

Description

This PR changes pnpm test to use Turborepo and adds support for Turborepo's cache in CI eslint and unit-tests.

The downside of this is not seeing all the green check marks when running tests locally, so I have left the option to run tests without Turborepo by running pnpm jest.

For CI I am using a GitHub Action that builds a local server (on port 9080) where Turborepo goes to look for the cache. To make it work, a TURBO_SERVER_TOKEN needs to be added to the repo.

If this is successful, we will be able to do the same for our e2e tests.

Resources:
Custom Remote Cache
TurboRepo with github artifacts

Steps to test the PR

pnpm test

Checklist

Quality: This PR builds and passes our npm test and works locally
Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not (if applicable)
Documentation: This PR includes updated/added documentation to user exposed functionality or configuration variables are added/changed or an explanation of why it does not(if applicable)

gitpod-io · 2022-04-11T14:55:07Z

humphd

This is awesome.

Should we do a follow-up to run a turbo-repo cache server on staging for local devs to leverage too? Or is this not needed locally?

humphd · 2022-04-11T15:12:20Z

.github/workflows/eslint-ci.yml

+        uses: felixmosh/turborepo-gh-artifacts@v1
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          server-token: ${{ secrets.TURBO_SERVER_TOKEN }}


What is this token supposed to look like? I'm reading the docs and I don't totally understand the point of it.

The answer is: it's an opaque secret token (can be anything), see felixmosh/turborepo-gh-artifacts#1 (comment)

I've added TURBO_SERVER_TOKEN to the repo.

menghif · 2022-04-11T15:47:48Z

This is awesome.

Should we do a follow-up to run a turbo-repo cache server on staging for local devs to leverage too? Or is this not needed locally?

I don't think it's needed for local dev since Turborepo keeps a local cache in a .turbo folder.

Also there is a new feature in version 1.2 to add signature verification:
https://turborepo.org/blog/turbo-1-2-0#cache-outputs-integrity-and-signature-verification
Is this something we should add?

humphd · 2022-04-11T15:48:14Z

.github/workflows/eslint-ci.yml

      - name: Install dependencies and run eslint
        run: |
          pnpm install
-          pnpm lint
+          pnpm turbo run lint --api="http://127.0.0.1:9080" --token="${{ secrets.TURBO_SERVER_TOKEN }}"


You can't access secrets.* directly in a called workflow. You have to pass the secret into the workflow. Checkout how I do it with the Docker username and password:

define the secret to be passed in https://github.com/Seneca-CDOT/telescope/blob/master/.github/workflows/docker-build-and-push.yml#L60-L63

pass it in https://github.com/Seneca-CDOT/telescope/blob/master/.github/workflows/continuous-delivery.yml#L14-L16

Do this for both of the workflow changes you're making.

humphd · 2022-04-11T15:49:11Z

This is awesome.
Should we do a follow-up to run a turbo-repo cache server on staging for local devs to leverage too? Or is this not needed locally?

I don't think it's needed for local dev since Turborepo keeps a local cache in a .turbo folder.

Also there is a new feature in version 1.2 to add signature verification: https://turborepo.org/blog/turbo-1-2-0#cache-outputs-integrity-and-signature-verification Is this something we should add?

We're not uploading them to a cache, right? I'm not sure we need this.

menghif · 2022-04-11T15:53:01Z

This is awesome.
Should we do a follow-up to run a turbo-repo cache server on staging for local devs to leverage too? Or is this not needed locally?

I don't think it's needed for local dev since Turborepo keeps a local cache in a .turbo folder.
Also there is a new feature in version 1.2 to add signature verification: https://turborepo.org/blog/turbo-1-2-0#cache-outputs-integrity-and-signature-verification Is this something we should add?

We're not uploading them to a cache, right? I'm not sure we need this.

Right, it would only be useful if we used an actual server.

.github/workflows/eslint-ci.yml

humphd · 2022-04-11T18:10:50Z

.github/workflows/unit-tests-ci.yml

@@ -2,6 +2,13 @@ name: Unit Tests Workflow

 on:
  workflow_call:
+    secrets:
+    github-token:


Same here, bump these in.

Just did. I don't understand why CI fails to start, I followed your docker example.

Invalid workflow file: .github/workflows/continuous-integration.yml#L22
The workflow is not valid. .github/workflows/continuous-integration.yml (Line: 22, Col: 21): Invalid secret, github-token is not defined in the referenced workflow. .github/workflows/continuous-integration.yml (Line: 23, Col: 24): Invalid secret, turborepo-token is not defined in the referenced workflow.

From https://github.com/Seneca-CDOT/telescope/actions/runs/2150363645

humphd

This looks right to me. NOTE: we don't run linked workflows from PRs, only what's on master, so we have to land this to see it happen.

humphd

I think it's uphappy with the GITHUB_TOKEN stuff

humphd · 2022-04-11T18:42:31Z

.github/workflows/continuous-integration.yml

@@ -18,10 +18,16 @@ jobs:
  # Make sure eslint passes
  eslint:
    uses: Seneca-CDOT/telescope/.github/workflows/eslint-ci.yml@master
+    secrets:
+      github-token: ${{ secrets.GITHUB_TOKEN }}


I think we can't pass this in (it's a synthesized token, not a real one), see https://github.community/t/how-to-pass-github-token-to-reusable-workflows/204905/6. What if you use {{ github.token }} below instead and get rid of this.

vercel bot deployed to Preview April 11, 2022 14:55 View deployment

menghif self-assigned this Apr 11, 2022

humphd reviewed Apr 11, 2022

View reviewed changes

humphd requested changes Apr 11, 2022

View reviewed changes

vercel bot deployed to Preview April 11, 2022 18:03 View deployment

humphd requested changes Apr 11, 2022

View reviewed changes

menghif force-pushed the issue-3351 branch from 4e5118e to 943042b Compare April 11, 2022 18:13

vercel bot deployed to Preview April 11, 2022 18:13 View deployment

humphd approved these changes Apr 11, 2022

View reviewed changes

humphd requested changes Apr 11, 2022

View reviewed changes

vercel bot deployed to Preview April 12, 2022 02:02 View deployment

menghif force-pushed the issue-3351 branch from 848a472 to 01dacc0 Compare April 12, 2022 02:08

vercel bot deployed to Preview April 12, 2022 02:08 View deployment

menghif force-pushed the issue-3351 branch from 01dacc0 to 14c5a56 Compare April 12, 2022 02:14

vercel bot deployed to Preview April 12, 2022 02:14 View deployment

DukeManh previously approved these changes Apr 12, 2022

View reviewed changes

DukeManh force-pushed the issue-3351 branch from 14c5a56 to b5cdf7f Compare April 12, 2022 04:23

vercel bot deployed to Preview April 12, 2022 04:23 View deployment

humphd approved these changes Apr 12, 2022

View reviewed changes

humphd dismissed DukeManh’s stale review via b5cdf7f April 12, 2022 11:47

JerryHue added this to the 3.0 Release milestone Apr 12, 2022

DukeManh approved these changes Apr 12, 2022

View reviewed changes

menghif added 4 commits April 12, 2022 08:12

Add test to Turborepo

8993174

Use Turborepo with cache in CI eslint and unit-tests

639d748

Pass secrets into workflow

9f20332

Use github.token instead of secrets.GITHUB_TOKEN

978ff33

change token name to server-token

90f86c9

menghif force-pushed the issue-3351 branch from b5cdf7f to 90f86c9 Compare April 12, 2022 12:12

vercel bot deployed to Preview April 12, 2022 12:12 View deployment

menghif merged commit 8bd3f84 into Seneca-CDOT:master Apr 12, 2022

humphd mentioned this pull request Apr 12, 2022

Add missing server-token to ci workflows #3461

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Turborepo for tests and add turbo cache for CI eslint and unit-tests #3458

Use Turborepo for tests and add turbo cache for CI eslint and unit-tests #3458

menghif commented Apr 11, 2022

gitpod-io bot commented Apr 11, 2022

humphd left a comment

humphd Apr 11, 2022

humphd Apr 11, 2022

humphd Apr 11, 2022

menghif commented Apr 11, 2022

humphd Apr 11, 2022

humphd commented Apr 11, 2022

menghif commented Apr 11, 2022

humphd Apr 11, 2022

menghif Apr 11, 2022

humphd Apr 11, 2022

humphd left a comment

humphd left a comment

humphd Apr 11, 2022

Use Turborepo for tests and add turbo cache for CI eslint and unit-tests #3458

Use Turborepo for tests and add turbo cache for CI eslint and unit-tests #3458

Conversation

menghif commented Apr 11, 2022

Issue This PR Addresses

Type of Change

Description

Steps to test the PR

Checklist

gitpod-io bot commented Apr 11, 2022

humphd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menghif commented Apr 11, 2022

Choose a reason for hiding this comment

humphd commented Apr 11, 2022

menghif commented Apr 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humphd left a comment

Choose a reason for hiding this comment

humphd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment