Docker image build always shows a resource to be updated

[discovery-NukulSharma]

Hi,
We are using github.com/pulumi/pulumi-docker/sdk/v2 v2.3.0 , and we are noticing a change is detected each time during build. This is leading to unnecessary redeployment.

This seems related to
#19
#28

However, we don't see it working, the PR #28, seems to have fixed only on typescript.

Do we have working solution, which avoids redeployment when application has not changed.

We fetch ECR credentials, and use it to push image, since ECR credentials are part of container def, it is detected as a change.
Sample code below..

username, password, proxyEndpoint := fetchECRRepositoryCredentials(ctx, repo.RegistryId)
image, err := docker.NewImage(ctx, imageName, &docker.ImageArgs{
	Build: &docker.DockerBuildArgs{
		Context:    dockerContextDir,
		Dockerfile: pulumi.Sprintf("%s%s", dockerContextDir, "Dockerfile"),
	},
	ImageName: pulumi.Sprintf("%s:latest", repo.RepositoryUrl),
	Registry: &docker.ImageRegistryArgs{
		Server:   proxyEndpoint,
		Username: username,
		Password: password,
	},
})

`

Appreciate if we can suggest a way forward pls.

pipeline detecting a change..

....

 ++ aws:ecs:TaskDefinition XYZ created replacement [diff: ~containerDefinitions]
 +- aws:ecs:TaskDefinition XYZ replacing [diff: ~containerDefinitions]
 +- aws:ecs:TaskDefinition XYZ replaced [diff: ~containerDefinitions]
 ~  aws:ecs:Service XYZ updating [diff: ~taskDefinition]
@ updating.............
 ~  aws:ecs:Service XYZ updated [diff: ~taskDefinition]
    aws:appautoscaling:Target XYZ  
    aws:appautoscaling:Policy XYZ 
 -- aws:ecs:TaskDefinition XYZ deleting original [diff: ~containerDefinitions]
 -- aws:ecs:TaskDefinition XYZ deleted original [diff: ~containerDefinitions]
    pulumi:pulumi:StackXYZ  1 message
 
Diagnostics:
  pulumi:pulumi:Stack (XYZ):
 
Resources:
    ~ 1 updated
    +-1 replaced
    2 changes. 8 unchanged

[leezen]

It looks like all the other language implementations of this actually do register the registry as an output (unlike the Node implementation)

[discovery-NukulSharma]

It looks like all the other language implementations of this actually do register the registry as an output (unlike the Node implementation)

Are there any plans for fix or do you have any suggestion if we have any way to tackle this?

[knvpk]

im also facing same issue.

[knvpk]

Any work arounds for now

[jkrusic19]

I too am facing this same issue in dotnet.

The Container Definition changes only because the image changes. I went and compared multiple json, Container Definition, outputs with the Pulumi Console and the only changing in the Task Defintion is the actual Image.

I currently do not have a work around other than the default deployment strategy for Fargate is a rolling update.

If there is any direction further I can assist with in debugging let me know.

[lukehoban]

@discovery-NukulSharma @pavankumarkatakam @jkrusic19 Does anyone have a repro of this they could share? I can't tell from the comments so far here exactly what is changing. The original code above shows creating an Image (but not the diff of the image) and the diff of a aws:ecs:TaskDefinition (but not the code for creating it). I'm having trouble reproducing anything unexpected from the snippets shared so far here.

[jkrusic19]

@lukehoban Next week I was going to spend some time on this, trying to determine if I can lock down any dependancies in my app project dockerfile, if they are not already locked down to specific version. Then run some tests with our app dockerfile and with a basic one.

[jkrusic19]

@lukehoban

Here is what I am seeing comparing local "up" to a remote CI "up."

Diff Output:

Locally:

• On subsequent pulumi up commands, I am not seeing an image needing to be updating. Therefore the container definition doesnt update, resulting is "No changes" which is expected.

Remote CI:

• On subsequent pulumi up command, I get a diff regardless if I build off the same CommitID or not. For our build agents we are using Azure Scale Sets, so the images are consistent and no one is installing anything adhoc. Therefore, I would expect the same results locally to the remote CI build agent, however it differs everytime only that of the Image of the ECS Task Definition/Container Definition.

I don't have a public repo for this since its in-house code. I was going to dumb it down to a simple nginx image and seeing if the CI system see's diffs. Our app dependancies are locked to the specific version so we are version locked.

I will keep testing things once I get some progress done for other items.

[lukehoban]

Thanks @jkrusic19 for the details. I believe the issue you are hitting here of different image SHAs from different machines is #23, and I believe that is largely tied to how Docker works across different environments.

I expect you could likely work around this by preparing your own tags (based on whatever condition defines when a new tag is needed) and using those as part of the image name.

[jkrusic19]

@lukehoban

So I think you guys are on the correct track into what is causing it, however I am going to throw you a wrench. With Azure Scale Sets or AWS AutoScaling groups they use the same exact AMI/VM. So, wouldn't/shouldn't it reproduce that same SHA?

[lukehoban]

So, wouldn't/shouldn't it reproduce that same SHA?

I am honestly not precisely sure what causes Docker to compute different hashes. It may even be related to your own Docker build having non-determintic elements to its build process. I'd love to have an isolated repo of this we could do a deeper dive on.

[jkrusic19]

@lukehoban

I am sharing my code with @pierskarsenbarg and a docker file for my lambda function. Maybe you guys can go off of that?

[jkrusic19]

@lukehoban

going back to your comment

I expect you could likely work around this by preparing your own tags (based on whatever condition defines when a new tag is needed) and using those as part of the image name.

All our builds are done via Docker, can you provide an example for this work-around? I have been thinking of a way to hash out the application code artifact into a zip and then need a way to know the previous state. If dotnet publish... -o /publish creates hash 1234 where should that get placed for the next run to validate against?

[jkrusic19]

@leezen @lukehoban

Can you guys describe this function more in-depth? Does this assist with this issue?
https://www.pulumi.com/docs/reference/pkg/docker/registryimage/

[leezen]

I've updated #132 with some of my findings.

[bombillazo]

Any update on this issue?

[AaronFriel]

@bombillazo @discovery-NukulSharma et al.: this issue is resolved with the new implementation of the Docker Image resource in v4! See our blog post for more info: https://www.pulumi.com/blog/build-images-50x-faster-docker-v4/