Make Image a real CustomResource

[lukehoban]

Today, the Image resource isn't a custom resource, and so doesn't actually participate in the CRUD lifecycle. This leads to several fairly major issues which we effectively cannot solve:

• Pulumi thinks it needs to replace ECS service and task definition even though no changes  #23 - Pulumi thinks it needs to replace ECS service and task definition even though no changes
• Diffs in Docker builds don't show up as diffs #131 - Diffs in Docker builds don't show up as diffs
• Don't build from scratch each time #14 - Don't build from scratch each time
• Avoid pushing Docker images when they haven't changed #32 - Avoid pushing Docker images when they haven't changed
• Fail early if Docker daemon is not available #88 - Fail early when Docker daemon is unavailable
• No docker:Image resource available in Pulumi YAML programs

We will need to move Image to being a CustomResource that can fully participate in the CRUD lifecycle. We will likely do this by re-implementing in Go either as part of the existing pulumi-terraform-bridge-based provider in this repo, or as a standalone Pulumi provider.

[discovery-NukulSharma]

do we have any plan , when can we expect this pls

[leezen]

I believe the RegistryImage custom resource that was recently added to this provider addresses some of the issues above. I'll need to validate that it does and if so, close out the relevant issues.

[leezen]

Evaluating RegistryImage, I found the following:

• Pulumi thinks it needs to replace ECS service and task definition even though no changes  #23 - Pulumi thinks it needs to replace ECS service and task definition even though no changes
• Don't build from scratch each time #14 - Don't build from scratch each time
• Avoid pushing Docker images when they haven't changed #32 - Avoid pushing Docker images when they haven't changed

The underlying source code of the provider suggests this is handled correctly via the StateFunc.
However, in my testing, I found this not to be the case (modifying the Dockerfile
or what it builds doesn't result in generating a change), so I think there may be a problem in the bridge we need to address.
That being said, feeding in the hash of the contents of the build context as the buildId has the desired effect (i.e. only builds/pushes on changes).

• Diffs in Docker builds don't show up as diffs #131 - Diffs in Docker builds don't show up as diffs

Relatedly, if there's a change in the build, this is picked up by the change in the hash now.

• Fail early if Docker daemon is not available #88 - Fail early when Docker daemon is unavailable

Using RegistryImage addresses this since it uses the actual provider, which upon initialization will fail to configure without a Docker daemon.

I think it's a fairly viable alternative to use RegistryImage over Image with the primary drawbacks being:

1. Needing to workaround the StateFunc issue by providing a hash via buildId instead of relying on the underlying StateFunc that computes the context hash.
2. There's no flow of the build output to pulumi which was the case with Image.

[leezen]

As an example of what I mean by computing a hash of the build context for buildId:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

import { hashElement } from "folder-hash";

// Create an ECR repo for our images.
const ecrRepo = new awsx.ecr.Repository("app");

// Create a Docker provider with the above registry setup.
const ecrCredentials = pulumi.output(aws.ecr.getAuthorizationToken());
const dockerHost = ecrCredentials.proxyEndpoint.apply(s => s.replace("https://", ""));
const dockerProvider = new docker.Provider("ecrDockerProvider", {
    registryAuth: [{
        "address": dockerHost,
        "password": ecrCredentials.password,
        "username": ecrCredentials.userName,
    }],
});

const buildDir = "../app";
const image = new docker.RegistryImage("image", {
    name: pulumi.interpolate`${ecrRepo.repository.repositoryUrl}:latest`,
    build: {
        buildId: hashElement(buildDir).then(h => h.toString()), // use the hash of the /app directory
        context: buildDir,
    },
}, { provider: dockerProvider });

[lukehoban]

There's no flow of the build output to pulumi which was the case with Image.

This feels like it could be a blocking issue for moving many usecases over. Is there any path we can imagine to supporting this?

[leezen]

One option for that would be to modify the upstream provider, which currently only uses the build response for error handling: https://github.com/kreuzwerker/terraform-provider-docker/blob/master/internal/provider/resource_docker_registry_image_funcs.go#L187 and pipe that output appropriately.

[dustinbrooks]

There's no flow of the build output to pulumi which was the case with Image.

This feels like it could be a blocking issue for moving many usecases over. Is there any path we can imagine to supporting this?

I was really hoping to leverage the RegistryImage, but yes, no output from Docker is a blocker for sure, apparently no one wants to run the full dockerfile locally first, just wait for the pipeline to tell us.

At this point it looks like we'll have to pull our image builds out of the Pulumi IaC and have the pipeline do them.

[benesch]

We're likely to tackle addressing some of these issues over at @MaterializeInc soon by building a custom, native Docker provider. I wanted to jot them down our requirements in case it helps inform a design here.

We need:

• A resource that builds an image locally and pushes it to a remote registry. (That's what this plugin calls docker.Image and docker.RemoteImage.) We don't need any of the other resources managed by this provider.
• Streaming output during pulumi preview and pulumi up, so that Pulumi doesn't appear to hang when it's really Docker being slow.
• Support for the Buildkit --platform option to build multi-arch or cross-arch images.
• Support for Buildkit inline caching in max mode so that the container registry can be used as a warm cache when pulumi up is run on a machine without a local Docker cache (e.g., CI).
• Stable IDs iff the build context is stable, even across machines. (Spurious rebuilds cause unnecessary and slow Kubernetes redeploys in our stack.)

I suspect those requirements are minimal enough that we can knock the provider out with just a day or two of work, since there's a lot of esoteric options we can ditch. Whatever we do we'll open source in case it's useful to other folks, or useful as a base for the official Docker provider.

[benesch]

As promised: https://github.com/benesch/pulumi-docker-buildkit. Hopefully it's useful to some other folks! It meets all the requirements I listed in my last comment.

Do note the disclaimer at the bottom of the README:

I plan to make minor bugfixes as necessary for our use of this provider at @MaterializeInc. I do not currently plan to bring it to feature parity with the official Docker provider, nor do I have the time to entertain such contributions. Sorry! I encourage you to either fork this repository or to use the ideas here to improve the official Pulumi Docker provider.

[JacobReynolds]

For others that are still running into various issues with Docker images in Pulumi, I wanted to share my approach. @leezen had a great option above but unfortunately for me an issue in the underlying provider made it not feasible. Trying to use @pulumi/docker.Image itself had the same issue as @pulumi/awsx.ecr.buildAndPushImage of building and pushing on every pulumi up.

With the recent release of @pulumi/command I've come up with an alright workaround, which I'm confident is an anti-pattern in some ways. Would love some feedback on if there's a way to integrate this approach in a more pulumi-centric way. There are still concerns about how to handle output as well as have intuitive error messages, but hopefully those will come alongside improvements in the command library.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import { local } from "@pulumi/command";
import { hashElement } from "folder-hash";

interface ImageArgs {
  context: string; //Relative from the base of the pulumi project
  buildArgs?: { [key: string]: pulumi.Output<string> | string };
  dockerfile?: string;
}

// This ComponentResource helps us not re-push docker images on every `pulumi up`
// There are built-in resources in pulumi for building and pushing images but a couple main issues were encounted:
//  @pulumi/awsx.ecr.buildAndPushImage rebuilds and repushes on every deploy, even if nothing changes
//  @pulumi/docker.Image does the same
//  @pulumi/docker.RegistryImage has an issue that the underlying provider does not respect file permissions, causing issues with certain builds, https://github.com/kreuzwerker/terraform-provider-docker/issues/293
//     Otherwise https://github.com/pulumi/pulumi-docker/issues/132#issuecomment-812234817 would be a nice solution
export class Image extends pulumi.ComponentResource {
  public imageName: pulumi.Output<string>;

  constructor(name: string, args: ImageArgs, opts: pulumi.ComponentResourceOptions = {}) {
    super("example:component:image", name, {}, opts);

    // Normalize the context to the base of the pulumi project
    const context = __dirname + args.context;

    // Hash the directory to be changed to help us identify if anytyhing has changed
    const hash = pulumi.output(hashElement(context, { encoding: "hex", folders: { exclude: ["node_modules", ".git"] } }).then((h: any) => h.hash));

    // Create the ECR repository and get credentials for it
    const repository = new awsx.ecr.Repository(name, {}, { parent: this });
    const ecrCredentials = pulumi.output(aws.ecr.getAuthorizationToken());
    const dockerHost = ecrCredentials.proxyEndpoint.apply((s) => s.replace("https://", ""));

    // Generate an image name that incorporates the hash of the directory
    this.imageName = pulumi.interpolate`${repository.repository.repositoryUrl}:${hash}`;

    let buildArgsOutputs: pulumi.Output<string>[] = [];
    for (const key in args.buildArgs) {
      buildArgsOutputs.push(pulumi.interpolate`--build-arg ${key}="${args.buildArgs[key]}"`);
    }
    let buildArgs = pulumi.all(buildArgsOutputs).apply((args) => args.join(" "));


    // Build and push locally (may have some requirements on your local environment, i.e. docker)
    // We use the bash `|| :` here because if there are concurrent builds the login command will fail since we're already logged in. I couldn't find any graceful ways to make this login work
    new local.Command(
      `${name}-docker-build-and-push`,
      {
        create: `(docker login -u $USERNAME -p $PASSWORD $ADDRESS || :) && docker build -t $NAME $BUILD_ARGS $CONTEXT -f $DOCKERFILE && docker push $NAME`,
        environment: {
          NAME: this.imageName,
          BUILD_ARGS: buildArgs,
          CONTEXT: context,
          ADDRESS: dockerHost,
          USERNAME: ecrCredentials.userName,
          PASSWORD: ecrCredentials.password,
          DOCKERFILE: args.dockerfile ? (args.dockerfile.indexOf("/") > -1 ? args.dockerfile : `${context}/${args.dockerfile}`) : `${context}/Dockerfile`,

        },
      },
      { parent: this, ignoreChanges: ["environment.USERNAME", "environment.PASSWORD"] }
    );

    this.registerOutputs({ imageName: this.imageName });
  }
}

[benesch]

FWIW, @JacobReynolds, that's pretty much exactly what https://github.com/MaterializeInc/pulumi-docker-buildkit does, except as a full-blown Pulumi provider so it can integrate Docker logs/errors with the Pulumi logging system.

I'd be curious to know if our provider would work for you, as I'd love to someday get its implementation upstreamed here! (No worries if it's against corporate policy or something to use third-party providers.)

[JacobReynolds]

@benesch glad to know I'm not alone :) Your provider was something I looked into but couldn't find a typescript SDK for, if that's available I'd love to give it a shot.

[benesch]

@benesch glad to know I'm not alone :) Your provider was something I looked into but couldn't find a typescript SDK for, if that's available I'd love to give it a shot.

Ahh, yeah, that makes sense. It's probably not too hard to get the JS SDK generation wired up...

[benesch]

I'm still checking whether this actually worked, but you can give https://www.npmjs.com/package/@materializeinc/pulumi-docker-buildkit v0.1.5 a whirl in the meantime!

[benesch]

Whew, well, that took much longer than I wanted, but I've validated that @materializeinc/pulumi-docker-buildkit works properly from TypeScript at v0.1.11.

[JacobReynolds]

Worked like a dream, that's awesome, thank you!

[benesch]

Woo, glad to hear it! Thanks for giving it a spin. 🙌🏽

[badokun]

@benesch my solution is in c#. Is there any mechanism to proxy thru to go? Not familiar with js and whether any interop is possible

[badokun]

I had a quick stab at it, but couldnt get it to work in c#. Likely to do with Outputs and not wiring up the dependency graph properly. It's just a rewrite of what @JacobReynolds did

public class CustomImage: ComponentResource
    {
        public Output<string> ImageName;
        
        private const string _rootAlphaCustomImageTypeName = "alpha:CustomImage";
        
        public CustomImage(string name, CustomImageArgs args, ComponentResourceOptions? options = null)
            : base(_rootAlphaCustomImageTypeName, name, options)
        {
            
            // Normalize the context to the base of the pulumi project
            var context = Path.GetFullPath(args.Context);

            var hash = GenerateHash(context);
            var dockerHost = args.ImageBuilderArgs.RegistryServer;
            
            ImageName = args.ImageBuilderArgs.RegistryServer.Apply(x => $"{x}/{name}:{hash}");

            var buildArgsOutputs = new List<Output<string>>();
            foreach (var kvp in args.BuildArgs)
            {
                buildArgsOutputs.Add(
                    Output.Create($"--build-arg {kvp.Key}={kvp.Value}")
                    );
            }
            
            var buildArgs = Output.All(buildArgsOutputs).Apply(a => string.Join(" ", a));

            var dockerFile = args.Dockerfile != null
                ? args.Dockerfile.IndexOf("/", StringComparison.Ordinal) > -1 ? args.Dockerfile : $"{context}/{args.Dockerfile}"
                : $"{context}/Dockerfile";

            // Build and push locally (may have some requirements on your local environment, i.e. docker)
            // We use the bash `|| :` here because if there are concurrent builds the login command will fail since
            // we're already logged in. I couldn't find any graceful ways to make this login work
            new Command($"{name}-docker-build-and-push",
                new CommandArgs()
                {
                    Create = "(docker login -u $USERNAME -p $PASSWORD $ADDRESS || :) && " +
                             "docker build -t $NAME $BUILD_ARGS $CONTEXT -f $DOCKERFILE && " +
                             "docker push $NAME",
                    Environment = new InputMap<string>()
                    {
                        { "NAME", ImageName },
                        { "BUILD_ARGS", buildArgs },
                        { "CONTEXT", context },
                        { "ADDRESS", dockerHost },
                        { "USERNAME", args.ImageBuilderArgs.RegistryUsername },
                        { "PASSWORD", args.ImageBuilderArgs.RegistryPassword },
                        { "DOCKERFILE", dockerFile }
                    }
                }, new CustomResourceOptions()
                {
                    Parent = this,
                    IgnoreChanges = new List<string>(){"environment.USERNAME", "environment.PASSWORD"}
                });
            
            RegisterOutputs(new Dictionary<string, object?>()
            {
                {"imageName", ImageName}
            });
        }

        private string GenerateHash(string context)
        {
            var allMD5Bytes = new List<byte>();
            var excludedDirectories = new[] { "bin", "obj" };
            var files = Directory.GetFiles(context, "*", SearchOption.AllDirectories);
            foreach (var fileName in files)
            {
                using var md5 = MD5.Create();
                var fileInfo = new FileInfo(fileName);
                if (excludedDirectories.Any(excludedDirectory =>
                        fileInfo.Directory != null && fileInfo.Directory.Name == excludedDirectory))
                    continue;
                
                using var stream = File.OpenRead(fileName);
                var md5Bytes = md5.ComputeHash(stream);
                
                allMD5Bytes.AddRange(md5Bytes);
            }

            using var hash = MD5.Create();
            var md5AllBytes = hash.ComputeHash(allMD5Bytes.ToArray());
            var result = BytesToHash(md5AllBytes);
            
            return result;
        }

        private string BytesToHash(IEnumerable<byte> md5Bytes)
        {
            return string.Join("", md5Bytes.Select(ba => ba.ToString("x2")));
        }
    }

    public class CustomImageArgs
    {
        public string Context { get; set; }
        public string? Dockerfile { get; set; }
        public ImageBuilderArgs ImageBuilderArgs { get; set; }
        public IDictionary<string, string> BuildArgs { get; set; }

    }

[andrekiba]

Hi, @badokun I'm also interested in a solution for C#, did you find it? @lukehoban @leezen Is there an eta to have the same capabilities also in the Pulumi.Docker provider for C#?

[badokun]

I ended up with a separate TypeScript project that builds my images. I use circle-ci so it's not really something I need to run myself.

Here's an example building two images in one project

import * as dockerBuildkit from "@materializeinc/pulumi-docker-buildkit";
import * as pulumi from "@pulumi/pulumi";

let stack = pulumi.getStack();
const cloudStack = new pulumi.StackReference(`xxxx/yyyy/${stack}`);

const registryInfo = {
    server: cloudStack.getOutput('RegistryServer'),
    username: cloudStack.getOutput('ClientId'),
    password: cloudStack.getOutput('ClientSecret')
};

let map: { [name: string]: object } = {};

const aaaImage = new dockerBuildkit.Image(
    "aaa-ts-image",
    {
        name: registryInfo.server.apply(n => `${n}/companyA/aaa:beta`),
        registry: registryInfo,
        context: '../../../src/CompanyA.aaa',
        dockerfile: 'Dockerfile'
    },
).repoDigest;

const bbbImage = new dockerBuildkit.Image(
    "bbb-ts-image",
    {
        name: registryInfo.server.apply(n => `${n}/companyA/bbb:beta`),
        registry: registryInfo,
        context: '../../../src/CompanyA.bbb',
        dockerfile: 'Dockerfile'
    },
).repoDigest;
 

map.aaa = aaaImage;
map.bbb = bbbImage;

export const Images = map;

[andrekiba]

@badokun thank you fo your reply, my problem is that I need to use docker images as input for other resources... Someway I have to link the two projects...

[badokun]

I had to do the same. By exporting the image as a stackoutput, your second project can reference that and you're all good

[andrekiba]

For others that are still running into various issues with Docker images in Pulumi, I wanted to share my approach. @leezen had a great option above but unfortunately for me an issue in the underlying provider made it not feasible. Trying to use @pulumi/docker.Image itself had the same issue as @pulumi/awsx.ecr.buildAndPushImage of building and pushing on every pulumi up.

With the recent release of @pulumi/command I've come up with an alright workaround, which I'm confident is an anti-pattern in some ways. Would love some feedback on if there's a way to integrate this approach in a more pulumi-centric way. There are still concerns about how to handle output as well as have intuitive error messages, but hopefully those will come alongside improvements in the command library.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import { local } from "@pulumi/command";
import { hashElement } from "folder-hash";

interface ImageArgs {
  context: string; //Relative from the base of the pulumi project
  buildArgs?: { [key: string]: pulumi.Output<string> | string };
  dockerfile?: string;
}

// This ComponentResource helps us not re-push docker images on every `pulumi up`
// There are built-in resources in pulumi for building and pushing images but a couple main issues were encounted:
//  @pulumi/awsx.ecr.buildAndPushImage rebuilds and repushes on every deploy, even if nothing changes
//  @pulumi/docker.Image does the same
//  @pulumi/docker.RegistryImage has an issue that the underlying provider does not respect file permissions, causing issues with certain builds, https://github.com/kreuzwerker/terraform-provider-docker/issues/293
//     Otherwise https://github.com/pulumi/pulumi-docker/issues/132#issuecomment-812234817 would be a nice solution
export class Image extends pulumi.ComponentResource {
  public imageName: pulumi.Output<string>;

  constructor(name: string, args: ImageArgs, opts: pulumi.ComponentResourceOptions = {}) {
    super("example:component:image", name, {}, opts);

    // Normalize the context to the base of the pulumi project
    const context = __dirname + args.context;

    // Hash the directory to be changed to help us identify if anytyhing has changed
    const hash = pulumi.output(hashElement(context, { encoding: "hex", folders: { exclude: ["node_modules", ".git"] } }).then((h: any) => h.hash));

    // Create the ECR repository and get credentials for it
    const repository = new awsx.ecr.Repository(name, {}, { parent: this });
    const ecrCredentials = pulumi.output(aws.ecr.getAuthorizationToken());
    const dockerHost = ecrCredentials.proxyEndpoint.apply((s) => s.replace("https://", ""));

    // Generate an image name that incorporates the hash of the directory
    this.imageName = pulumi.interpolate`${repository.repository.repositoryUrl}:${hash}`;

    let buildArgsOutputs: pulumi.Output<string>[] = [];
    for (const key in args.buildArgs) {
      buildArgsOutputs.push(pulumi.interpolate`--build-arg ${key}="${args.buildArgs[key]}"`);
    }
    let buildArgs = pulumi.all(buildArgsOutputs).apply((args) => args.join(" "));


    // Build and push locally (may have some requirements on your local environment, i.e. docker)
    // We use the bash `|| :` here because if there are concurrent builds the login command will fail since we're already logged in. I couldn't find any graceful ways to make this login work
    new local.Command(
      `${name}-docker-build-and-push`,
      {
        create: `(docker login -u $USERNAME -p $PASSWORD $ADDRESS || :) && docker build -t $NAME $BUILD_ARGS $CONTEXT -f $DOCKERFILE && docker push $NAME`,
        environment: {
          NAME: this.imageName,
          BUILD_ARGS: buildArgs,
          CONTEXT: context,
          ADDRESS: dockerHost,
          USERNAME: ecrCredentials.userName,
          PASSWORD: ecrCredentials.password,
          DOCKERFILE: args.dockerfile ? (args.dockerfile.indexOf("/") > -1 ? args.dockerfile : `${context}/${args.dockerfile}`) : `${context}/Dockerfile`,

        },
      },
      { parent: this, ignoreChanges: ["environment.USERNAME", "environment.PASSWORD"] }
    );

    this.registerOutputs({ imageName: this.imageName });
  }
}

@JacobReynolds @lukehoban were you able to run the docker commands inside the pulumi command without problems? Trying this I always have “exit code 1” without the possibility to understand what is wrong. Could you please help me?

[AaronFriel]

This issue is resolved with the new implementation of the Docker Image resource in v4! See our blog post for more info: https://www.pulumi.com/blog/build-images-50x-faster-docker-v4/