Add support for SHA-256 Git commit IDs

[tianon]

This also adds a few (mostly edge-case) tests for the gitutil.IsCommitSHA function.

(This is a follow-up to #5441 (comment))

[tianon]

To be clear: if the SHA-256 addition here is too controversial, I'm happy to split out the added test (and adjust it to move the SHA-256 expected-success into the expected-failure group). ❤️

[tonistiigi]

Is it possible to get an actual integration test in https://github.com/moby/buildkit/blob/master/source/git/source_test.go for this functionality?

[tianon]

Thanks for the pointer! I will take a look and see what I can figure out!

[tianon]

Recording for posterity: I was hoping I could lean on git/git@9ae702f to initialize as sha1 but allow querying/use via sha256 without splitting the code, but that was added in Git 2.45 and Debian Bookworm only has 2.39, so it doesn't seem super reasonable to lean on that and I'll have to split the git init code to do a full --object-format=sha256 version (or have a subset that's used just for testing sha256 fetch-by-commit). Either way there are existing tests hard-coded to the 40 character length too, so I'm in for more than I bargained for but it's important work because future Git versions might break these assumptions by default someday. 😂

[tianon]

Gah yep, this is almost immediately touching things wider than I intended; I've got tests failing because gitSourceHandler doesn't support SHA-256, which makes sense but also can't be fixed in a reasonable way that can support both SHA-1 and SHA-256 concurrently without git/git@9ae702f (extensions.compatObjectFormat, Git 2.45+), so I might need to add what I've got so far, move this to draft, and split out the new test into a separate PR without SHA-256. 😭

[tianon]

To be clear: I'm officially above my ability to continue here -- we need either a newer minimum Git version somehow (which doesn't seem reasonable) or a larger refactoring to encode --object-format=sha256 more directly / explicitly within BuildKit itself, so I'd be totally on board with someone else stealing/carrying this if they think they can overcome that. 👍

[tianon]

(Rebased, especially for #5476)

FWIW, GitLab has (public!) support for SHA-256 repositories for several months now: https://about.gitlab.com/blog/2024/08/19/gitlab-now-supports-sha256-repositories/

I created one to help make this easier to test: https://gitlab.com/tianon/test-sha256

Here's what happens when one tries to build it with current buildkit (c9a17ff):

$ docker buildx build --builder master https://gitlab.com/tianon/test-sha256.git
#0 building with "master" instance using docker-container driver

#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
#1 ERROR: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4"
------
 > [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
------
ERROR: failed to solve: failed to read dockerfile: failed to load cache key: invalid commit sha "6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4"

And with this PR (b160376, specifically):

$ docker buildx build --builder pr-5471 https://gitlab.com/tianon/test-sha256.git
#0 building with "pr-5471" instance using docker-container driver

#1 [internal] load git source https://gitlab.com/tianon/test-sha256.git
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
fatal: mismatched algorithms: client sha1; server sha256
#1 ERROR: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128
------
 > [internal] load git source https://gitlab.com/tianon/test-sha256.git:
Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	refs/heads/main
ref: refs/heads/main	HEAD
6927b3670b78e949968e1b7b4dc04553b1668bef2e17ec1e7f83c6596d59a8f4	HEAD
fatal: mismatched algorithms: client sha1; server sha256
------
ERROR: failed to solve: failed to read dockerfile: failed to fetch remote https://gitlab.com/tianon/test-sha256.git: git stderr:
fatal: mismatched algorithms: client sha1; server sha256
: exit status 128

So there's clearly still work to do, and it's very related to my struggles with the integration tests -- however, as noted above, I'm beyond my ability to continue here, so someone else will have to do the refactoring work / integration from here. 🙇

(Also, I think it's worth noting that the importance of making this work is only going to increase over time -- see also https://github.blog/open-source/git/highlights-from-git-2-45/#preliminary-support-for-sha-1-and-sha-256-interoperability)