-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for SHA-256 Git commit IDs #5471
base: master
Are you sure you want to change the base?
Conversation
To be clear: if the SHA-256 addition here is too controversial, I'm happy to split out the added test (and adjust it to move the SHA-256 expected-success into the expected-failure group). ❤️ |
Is it possible to get an actual integration test in https://github.com/moby/buildkit/blob/master/source/git/source_test.go for this functionality? |
Thanks for the pointer! I will take a look and see what I can figure out! |
Recording for posterity: I was hoping I could lean on git/git@9ae702f to initialize as sha1 but allow querying/use via sha256 without splitting the code, but that was added in Git 2.45 and Debian Bookworm only has 2.39, so it doesn't seem super reasonable to lean on that and I'll have to split the |
Gah yep, this is almost immediately touching things wider than I intended; I've got tests failing because |
To be clear: I'm officially above my ability to continue here -- we need either a newer minimum Git version somehow (which doesn't seem reasonable) or a larger refactoring to encode |
This also adds a few (mostly edge-case) tests for the `gitutil.IsCommitSHA` function. Signed-off-by: Tianon Gravi <[email protected]>
This fails in `gitSourceHandler` because it can't handle *both* SHA-1 and SHA-256 before Git 2.45 (see PR comments), and there's not a simple way to fix that without newer Git or larger refactoring.
(Rebased, especially for #5476) FWIW, GitLab has (public!) support for SHA-256 repositories for several months now: https://about.gitlab.com/blog/2024/08/19/gitlab-now-supports-sha256-repositories/ I created one to help make this easier to test: https://gitlab.com/tianon/test-sha256 Here's what happens when one tries to build it with current buildkit (c9a17ff): $ docker buildx build --builder master https://gitlab.com/tianon/test-sha256.git
#0 building with "master" instance using docker-container driver
#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 refs/heads/main
#1 ERROR: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4"
------
> [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 refs/heads/main
------
ERROR: failed to solve: failed to read dockerfile: failed to load cache key: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4" And with this PR (b160376, specifically): $ docker buildx build --builder pr-5471 https://gitlab.com/tianon/test-sha256.git
#0 building with "pr-5471" instance using docker-container driver
#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 refs/heads/main
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
fatal: mismatched algorithms: client sha1; server sha256
#1 ERROR: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128
------
> [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 refs/heads/main
ref: refs/heads/main HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4 HEAD
fatal: mismatched algorithms: client sha1; server sha256
------
ERROR: failed to solve: failed to read dockerfile: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128 So there's clearly still work to do, and it's very related to my struggles with the integration tests -- however, as noted above, I'm beyond my ability to continue here, so someone else will have to do the refactoring work / integration from here. 🙇 (Also, I think it's worth noting that the importance of making this work is only going to increase over time -- see also https://github.blog/open-source/git/highlights-from-git-2-45/#preliminary-support-for-sha-1-and-sha-256-interoperability) |
This also adds a few (mostly edge-case) tests for the
gitutil.IsCommitSHA
function.(This is a follow-up to #5441 (comment))