Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for SHA-256 Git commit IDs #5471

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

tianon
Copy link
Member

@tianon tianon commented Oct 30, 2024

This also adds a few (mostly edge-case) tests for the gitutil.IsCommitSHA function.

(This is a follow-up to #5441 (comment))

@tianon
Copy link
Member Author

tianon commented Oct 30, 2024

To be clear: if the SHA-256 addition here is too controversial, I'm happy to split out the added test (and adjust it to move the SHA-256 expected-success into the expected-failure group). ❤️

@tonistiigi
Copy link
Member

Is it possible to get an actual integration test in https://github.com/moby/buildkit/blob/master/source/git/source_test.go for this functionality?

@tianon
Copy link
Member Author

tianon commented Oct 31, 2024

Thanks for the pointer! I will take a look and see what I can figure out!

@tianon
Copy link
Member Author

tianon commented Oct 31, 2024

Recording for posterity: I was hoping I could lean on git/git@9ae702f to initialize as sha1 but allow querying/use via sha256 without splitting the code, but that was added in Git 2.45 and Debian Bookworm only has 2.39, so it doesn't seem super reasonable to lean on that and I'll have to split the git init code to do a full --object-format=sha256 version (or have a subset that's used just for testing sha256 fetch-by-commit). Either way there are existing tests hard-coded to the 40 character length too, so I'm in for more than I bargained for but it's important work because future Git versions might break these assumptions by default someday. 😂

@tianon
Copy link
Member Author

tianon commented Oct 31, 2024

Gah yep, this is almost immediately touching things wider than I intended; I've got tests failing because gitSourceHandler doesn't support SHA-256, which makes sense but also can't be fixed in a reasonable way that can support both SHA-1 and SHA-256 concurrently without git/git@9ae702f (extensions.compatObjectFormat, Git 2.45+), so I might need to add what I've got so far, move this to draft, and split out the new test into a separate PR without SHA-256. 😭

@tianon tianon marked this pull request as draft October 31, 2024 21:45
@tianon
Copy link
Member Author

tianon commented Oct 31, 2024

To be clear: I'm officially above my ability to continue here -- we need either a newer minimum Git version somehow (which doesn't seem reasonable) or a larger refactoring to encode --object-format=sha256 more directly / explicitly within BuildKit itself, so I'd be totally on board with someone else stealing/carrying this if they think they can overcome that. 👍

This also adds a few (mostly edge-case) tests for the `gitutil.IsCommitSHA` function.

Signed-off-by: Tianon Gravi <[email protected]>
This fails in `gitSourceHandler` because it can't handle *both* SHA-1 and SHA-256 before Git 2.45 (see PR comments), and there's not a simple way to fix that without newer Git or larger refactoring.
@tianon
Copy link
Member Author

tianon commented Nov 11, 2024

(Rebased, especially for #5476)

FWIW, GitLab has (public!) support for SHA-256 repositories for several months now: https://about.gitlab.com/blog/2024/08/19/gitlab-now-supports-sha256-repositories/

I created one to help make this easier to test: https://gitlab.com/tianon/test-sha256

Here's what happens when one tries to build it with current buildkit (c9a17ff):

$ docker buildx build --builder master https://gitlab.com/tianon/test-sha256.git
#0 building with "master" instance using docker-container driver

#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
#1 ERROR: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4"
------
 > [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
------
ERROR: failed to solve: failed to read dockerfile: failed to load cache key: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4"

And with this PR (b160376, specifically):

$ docker buildx build --builder pr-5471 https://gitlab.com/tianon/test-sha256.git
#0 building with "pr-5471" instance using docker-container driver

#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
fatal: mismatched algorithms: client sha1; server sha256
#1 ERROR: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128
------
 > [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
fatal: mismatched algorithms: client sha1; server sha256
------
ERROR: failed to solve: failed to read dockerfile: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128

So there's clearly still work to do, and it's very related to my struggles with the integration tests -- however, as noted above, I'm beyond my ability to continue here, so someone else will have to do the refactoring work / integration from here. 🙇

(Also, I think it's worth noting that the importance of making this work is only going to increase over time -- see also https://github.blog/open-source/git/highlights-from-git-2-45/#preliminary-support-for-sha-1-and-sha-256-interoperability)

@thompson-shaun thompson-shaun added this to the v0.future milestone Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants