Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp release automation #1250

Merged
merged 26 commits into from
Aug 14, 2023
Merged

Revamp release automation #1250

merged 26 commits into from
Aug 14, 2023

Conversation

hiddeco
Copy link
Member

@hiddeco hiddeco commented Jul 11, 2023

This pull request changes the release automation so that it uses GoReleaser.

The configuration for GoReleaser is in the file .goreleaser.yaml. The configuration for the GitHub Actions workflow is in the file release.yml.

This configuration is quite sophisticated, and ensures at least the following:

  • The release is built for multiple platforms and architectures, including Linux, macOS, and Windows, and for both AMD64 and ARM64.
  • The release includes multiple packages in Debian and RPM formats.
  • For every binary, a corresponding SBOM is generated and published.
  • For all binaries, a checksums file is generated and signed using Cosign with GitHub OIDC.
  • Both Debian and Alpine Docker multi-arch images are built and pushed to GitHub Container Registry and Quay.io.
  • The container images are signed using Cosign with GitHub OIDC.
  • SLSA provenance metadata is generated for release artifacts and container images.

⚠️ Notable changes

  • The previous Dockerfile and Dockerfile.alpine have been removed in favor of Dockerfiles specifically crafted for GoReleaser in .release/. These Dockerfiles need a specially crafted build context (with pretty much just the binary in it), and do not work using a normal docker build ....
  • While the release artifacts have the intention to be as backwards compatible as possible in terms of file names, etc. Your mileage may vary here, more specifically the .rpm and .deb artifacts.
  • The .darwin artifact is now a "fat binary".

👀 Preview (on fork)

🧑‍🏭 Preview workflow (on fork)

🖥️ Testing locally

Run make release-snapshot and view dist/ contents.

@hiddeco hiddeco force-pushed the release-automation branch 4 times, most recently from 2a71e05 to 9da6f02 Compare July 11, 2023 22:17
@hiddeco hiddeco added this to the v3.8.0 milestone Jul 12, 2023
@hiddeco hiddeco force-pushed the release-automation branch from 9da6f02 to d80f687 Compare July 12, 2023 21:51
@hiddeco hiddeco force-pushed the release-automation branch 9 times, most recently from 421b4e8 to 951b428 Compare August 2, 2023 22:35
@hiddeco hiddeco force-pushed the release-automation branch 2 times, most recently from 3874a58 to 3517a21 Compare August 8, 2023 23:31
@hiddeco
Copy link
Member Author

hiddeco commented Aug 9, 2023

@getsops/maintainers this is ready for an early review. I need to update the PR message to include all details, but also really need to catch some sleep at the moment.

The thing to focus on in the preview is the file formats of the artifacts, in comparison to https://github.com/getsops/sops/releases/tag/v3.7.3. Most of them should be fully backwards compatible (the binaries), but for the package formats I have reached the absolute limits of what I deemed possible.

In addition, it includes all the niceness of a checksum file signed with Cosign, SBOMs for the binaries (but not containers, which include a 1:1 copy of the binary — we can add this later), and SLSA provenance for all artifacts uploaded to GitHub and the container images themselves.

To get an idea of the changelog which would normally be included, run:

gh api \
  --method POST \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  /repos/getsops/sops/releases/generate-notes \
  -f tag_name='v3.8.0' \
  -f target_commitish='main' \
  -f previous_tag_name='v3.7.2' | jq -r .body

Based on the output, I think the current header configuration actually should move to footer.

@hiddeco hiddeco marked this pull request as ready for review August 9, 2023 00:05
@hiddeco hiddeco force-pushed the release-automation branch from 3c2999f to 4439cc9 Compare August 9, 2023 08:34
@hiddeco hiddeco force-pushed the release-automation branch 5 times, most recently from 99d6b5e to d5a8a3a Compare August 10, 2023 22:14
hiddeco added 26 commits August 14, 2023 22:37
This adds the base for releasing using GoReleaser going forward in a
backwards compatible manner, which means:

- Publishing of artifacts in the same formats as previous releases
- Publishing of RPM and deb artifacts in the same formats as previous
  releases (although the metadata may need a bit of tweaking)

In addition, it includes:

- SBOM inclusion per binary artifact

It still needs work around:

- Artifact signing
- SLSA compliance
- Docker images
- GitHub release
- Changelog generation
- GitHub Action workflow

Signed-off-by: Hidde Beydals <[email protected]>
GoReleaser requires specifically crafted Dockerfiles as the build
context is dynamically constructed.

For more information, refer to https://goreleaser.com/errors/docker-build/#do
and other documentation around Docker image templates and manifests.

Signed-off-by: Hidde Beydals <[email protected]>
This appears to be the best option at present to e.g. celebrate new
contributors while also allowing things to be grouped by pull request
label.

For more information, see xrefs in patch.

Signed-off-by: Hidde Beydals <[email protected]>
This still needs further configuration of at least the `.header` field.

Signed-off-by: Hidde Beydals <[email protected]>
Plus a tiny nit to not have to pass `--yes` to Cosign everywhere, and
enabling of size reporting.

Signed-off-by: Hidde Beydals <[email protected]>
These are no longer required, as they are now handled by GoReleaser
or no longer under our control (`make_download_page.sh`).

Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
Which in turn solves the generation of the checksum file.

Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
These were only part of the release process, and now continue to exist
in `.release/*`.

Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
As it has been replaced with GoReleaser.

Signed-off-by: Hidde Beydals <[email protected]>
This allows you to run the release locally, without publishing or
signing, against the current state of the repository.

There are some more improvements I would like to make to the
`Makefile` e.g., the deprecation of `golint` and the introduction of a
`help` target. But they are out of scope for the current things I am
working on.

Signed-off-by: Hidde Beydals <[email protected]>
Which now allows us to set the `mod_timestamp` on universal binaries.

Signed-off-by: Hidde Beydals <[email protected]>
- Describe difference between Debian (slim) and Alpine image
- Add `-o text` flag to `cosign verify` example to print readable text
  instead of JSON blob
- Fix typo in one of the `ghcr.io` domains
- Use correct OCI annotation for image "title"

Signed-off-by: Hidde Beydals <[email protected]>
@hiddeco hiddeco force-pushed the release-automation branch from 68aa071 to 79bbb22 Compare August 14, 2023 20:37
@hiddeco hiddeco merged commit 7fd5751 into main Aug 14, 2023
@hiddeco hiddeco deleted the release-automation branch August 14, 2023 20:46
@hiddeco hiddeco mentioned this pull request Aug 25, 2023
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants