Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for multi-arch OCI image indexes in Bazel builder #9217

Open
ar3s3ru opened this issue Dec 12, 2023 · 2 comments · May be fixed by #9219
Open

Add support for multi-arch OCI image indexes in Bazel builder #9217

ar3s3ru opened this issue Dec 12, 2023 · 2 comments · May be fixed by #9219
Labels
build/bazel priority/p2 May take a couple of releases

Comments

@ar3s3ru
Copy link
Contributor

ar3s3ru commented Dec 12, 2023

I am working in a Bazel workspace that uses rules_oci to build image tarballs, and Skaffold to perform the build+deploy steps.

As I'm working on an M1 (and minikube) but deploying on a remote k8s cluster based on linux-amd64, the images I'm building are multi-arch.

When running skaffold dev, it fails with the following error:

skaffold dev -p local --port-forward
Generating tags...
 - api -> api:v0.31.1-6-g1781fcd-dirty
Checking cache...
 - api: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [api]...
Target platforms: [linux/arm64]
Loading: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //backend/apps/api:image.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //backend/apps/api:image.tar (46 packages loaded, 1470 targets configured).
INFO: Found 1 target...
[5 / 16] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //backend/apps/api:image.tar up-to-date:
  bazel-bin/backend/apps/api/image.tar/tarball.tar
INFO: Elapsed time: 0.771s, Critical Path: 0.43s
INFO: 3 processes: 2 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 3 total actions
Cleaning up...
 - No resources found
Error: uninstall: Release not loaded: postgres: release: not found
Error: uninstall: Release not loaded: redis: release: not found
Error: uninstall: Release not loaded: jaeger: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
build [api] failed: loading manifest from tarball failed: file manifest.json not found in tar

Here is an excerpt of the BUILD.bazel file, note how format = "oci" in oci_tarball:

BUILD.bazel 
go_binary(
    name = "api",
    embed = [":api_lib"],
    visibility = ["//visibility:public"],
)

linux_amd64_binary = "api_linux-amd64".format(app_name)
linux_amd64_layer = "{}_layer".format(linux_amd64_binary)
linux_amd64_image = "{}_image".format(linux_amd64_binary)

go_cross_binary(
    name = linux_amd64_binary,
    platform = "@rules_go//go/toolchain:linux_amd64",
    target = target,
)

pkg_tar(
    name = linux_amd64_layer,
    srcs = [linux_amd64_binary],
)

oci_image(
    name = linux_amd64_image,
    base = base,
    entrypoint = ["/api"],
    tars = [linux_amd64_layer],
)

linux_arm64_binary = "api_linux-arm64"
linux_arm64_layer = "{}_layer".format(linux_arm64_binary)
linux_arm64_image = "{}_image".format(linux_arm64_binary)

go_cross_binary(
    name = linux_arm64_binary,
    platform = "@rules_go//go/toolchain:linux_arm64",
    target = target,
)

pkg_tar(
    name = linux_arm64_layer,
    srcs = [linux_arm64_binary],
)

oci_image(
    name = linux_arm64_image,
    base = base,
    entrypoint = ["/api"],
    tars = [linux_arm64_layer],
)

oci_image_index(
    name = "image",
    images = [
        linux_amd64_image,
        linux_arm64_image,
    ],
)

oci_tarball(
    name = "image.tar",
    format = "oci",
    image = ":image",
    repo_tags = ["api:latest"],
)

This is happening due to the following section:

skaffold/pkg/skaffold/build/bazel/build.go

Lines 93 to 96 in 1255d6c

func (b *Builder) loadImage(ctx context.Context, out io.Writer, tarPath string, tag string) (string, error) {
manifest, err := tarball.LoadManifest(func() (io.ReadCloser, error) {
return os.Open(tarPath)
})

tarball.LoadManifest looks for the manifest.json file in the tarball built by Bazel, but won't find it since OCI image indexes use an index.json file instead: https://github.com/google/go-containerregistry/blob/4fdaa32ee934cd178b6eb41b3096419a52ef426a/pkg/v1/tarball/image.go#L74-L79

go-containerregistry already exposes a package to load the index.json file: layout.ImageIndexFromPath

RepoTags can be deducted using the stategy explained here: opencontainers/image-spec#796
@ericzzzzzzz ericzzzzzzz added build/bazel priority/p2 May take a couple of releases labels Dec 12, 2023
@ar3s3ru ar3s3ru linked a pull request Dec 12, 2023 that will close this issue
Draft
@ar3s3ru
Copy link
Contributor Author

ar3s3ru commented Dec 12, 2023

I'm giving it a try in #9219.

I managed to make it load the index.json manifest, but performing the docker load operation locally is failing with the following error:

build [api] failed: loading image into docker daemon: reading from image load response: open /var/lib/docker/tmp/docker-import-3424243076/blobs/json: no such file or directory

It appears to be due to the fact that the Docker Engine API expects a very specific format for the tarball: https://docs.docker.com/engine/api/v1.43/#tag/Image/operation/ImageLoad

@ar3s3ru ar3s3ru mentioned this issue Dec 12, 2023
Merged
@aran
Copy link
Contributor

aran commented Dec 14, 2023

This would be really useful. Looks like Docker Engine doesn't support OCI-format multi-arch tarballs out of the box, rules_oci doesn't produce Docker-format multi-arch tarballs, and skaffold doesn't currently support alternative container APIs like containerd. (https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/docker/image.go would need to be generalized). Might need a new bazel rule something like docker_tarball to make tarballs in the format docker API expects, or generalize skaffold to be able to communicate with a containerd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build/bazel priority/p2 May take a couple of releases
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(build/bazel): support OCI ImageIndex tarballs ar3s3ru/skaffold
3 participants
@aran @ar3s3ru @ericzzzzzzz