Listing tags in JFrog Artifactory may fail

[SSW-SCIENTIFIC]

Hello team,

I want to notify I have encountered en error on skopeo v1.15.1 with multiarch images on JFrog Artifactory.

JFrog Artifactory is an artifact management tool that can be behaved as container image registry.
However, for multiarch images, it violates some behaviors related to tags.

For example,

1. once pushed an multiarch image multi-arch:v1.0.0, which refers sha256:aaaa...aaaa as a amd64, and sha256:bbbb...bbbb as a arm64, to Artifactory,
2. Artifactory expose sha256:aaaa...aaaa and sha256:bbbb...bbbb as tags, not only multi-arch:v1.0.0,
3. and sha256:aaaa...aaaa and sha256:bbbb...bbbb are OCI incompliant tags because they contains :,

So finally we got error on skopeo inspect with time="2024-06-03T09:26:46Z" level=fatal msg="Error determining repository tags: registry returned invalid tag \"sha256:aaaa...aaaa": invalid tag format".

This is quite heavy impact issue even if the Artifactory violates OCI specifications, we can no longer use skopeo v1.15.1 or later with Artifactory.

Can you consider that skopeo simply ignores the invalid tags to keep compatibility?
And we will also contact to JFrog support to improve the behavior above.

[marcaurele]

We are having the same problem on our side with skopeo version 1.15.1. It used to work correctly.

[mtrmac]

Thanks for your report. Could you test with containers/image#2440 and confirm that that is sufficient to list all tags without failure, please?

[SSW-SCIENTIFIC]

@mtrmac

git clone https://github.com/containers/skopeo.git && cd skopeo
git checkout release-1.15
go get -d github.com/mtrmac/image/v5@jfrog-invalid-tag
go mod edit -replace=github.com/containers/image/v5=github.com/mtrmac/image/[email protected]
make vendor
go mod tidy
DISABLE_DOCS=1 make

and then run bin/skopeo.
It works good for me (ignoring tags with format sha256:... without errors).

Thank you!

[carlosmmatos]

@mtrmac I've also confirmed this patch works by ignoring the sha256: values when listing tags.

[Romain-Geissler-1A]

@TomSweeneyRedHat I unfortunately hit this issue as well.

In term of releasing, I see that you are about to cut some releases of containers/image, does it mean there will be a point release 1.15.X of skopeo with a vendored version of containers/image ? If not, would you accept some patch backport sent directly into the fedora package to fix at least this issue for fedora users using Artifactory ?

[mtrmac]

@Romain-Geissler-1A I think we should make a new Skopeo upstream version (after #2451 merges here).

[Romain-Geissler-1A]

Thanks for the update.

FYI, since on our side we are also paying customers on JFrog side, I have raised a ticket there and they just replied me this:

Upon reviewing this issue, we found it to be a previously reported problem. A Jira ticket with the internal ID [RTDEV-43592] has already been raised. This ticket describes the issue: “Docker tags are displaying SHA256 for multi-architecture images when we list the tags,” and it has been marked for our product team's review. Please note that [RTDEV-43592] is an internal Jira ticket, so you might not be able to access it

On our side, we are as blind as you on the status of this internal Jfrog ticket, so the only thing I can say is that Jfrog acknowledges they don't fully follow the OCI registry spec on this specific point and for now there exist no fixed Artifatory release.

[SSW-SCIENTIFIC]

Sorry for late response.
I have received response from JFrog support, they said that Docker buildx tool pushes tags in a format of sha256:*** for multi-arch images, so not only JFrog Artifactory but also DockerHub store tags with format sha256:***.
By contrast to JFrog Artifactory, DockerHub does not returns tags with that format on TagList API call.

(I feel that they will not fix this behavior even if this is not compliant with neither OCI nor Docker specifications.)

[tomarie]

Hi @SSW-SCIENTIFIC & @marcaurele,

I'm Tom, the Product Manager leading Docker & OCI package integrations at @jfrog Artifactory. Thank you for bringing this important issue to our attention and I apologize for not responding sooner.

Firstly, I would like to assure you that we are actively addressing this issue as we speak and we will be ready shortly with a patch that aligns our /tags/list API response with Docker & OCI specifications by omitting non-readable tags from the response.

I’d like to take this opportunity to share two points that are very important to us at @jfrog which are the amazing use patterns we see our community and customers are doing with technologies like OCI and secondly the fact that specifications and community usages are evolving.
When we originally designed our /tags/list API we came across issues where our customers were looking for ways to see *All tags in a repository.
This made sense to us at the time since we believe we should provide trust & transparency always. We also witnessed changes like the one made to buildx a while back where multi-arch images stopped being tagged as Arch/OS and shifted to Digests, making them generic for future OCI use cases which pointed us a little further in the direction of responding with all the content.

The latest change in the containers project reflects OCI's evolution by now enforcing a rule that tags cannot include certain characters (colon specifically). Since Artifactory does not intervene with the Clients tagging process the existence of these characters was not enforced previously and should be looked into by our friends in the client tool communities.
(Please note that it is highly probable that other tools out there still tag images in a manner that now violates this validation).

At the time we did our best to balance between the motivations of Specification and Registry alignment and maintaining trust and transparency and until this last change we had no issues.

With that being said, we truly understand these pains and we embrace change and want to continue to do our best to accommodate our communities & customer use cases and this is why we are of course making these changes.

Furthermore, we will be actively reaching out to our developer community to promote this validation issue on the client level in hopes to prevent invalid characters even reaching a registry.

Lastly, I would like to thank you all for your collaboration and positive discussion and we’re looking forward to collaborating again in the future

Best regards,
Tom

[mtrmac]

The latest change in the containers project reflects OCI's evolution by now enforcing a rule that tags cannot include certain characters (colon specifically). Since Artifactory does not intervene with the Clients tagging process the existence of these characters was not enforced previously and should be looked into by our friends in the client tool communities.
(Please note that it is highly probable that other tools out there still tag images in a manner that now violates this validation).

Thanks for the information. So there is a risk of the “list tags” endpoint returning values with other formats not matching the <reference> format of https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests ?

Right now the c/image code only specifically ignores values that match the digest format. Strings that match neither the tag nor the digest format would still cause a failure to parse the list output. (Such strings could never be referenced, for reading or writing, using the c/image code, either way — but their mere presence might prevent listing other tag values.)

At this point we have no records of such unusual tag values happening; but the validation is new.

[mtrmac]

A fix was now released in Skopeo 1.15.2, I apologize for the time necessary to run that process.