Avoid pushing Docker images when they haven't changed

[joeduffy]

It would be nice if we could use the digest to avoid re-pushing Docker images that didn't actually change. This gets cached by the server and we avoid pushing, but it takes many seconds of watching Waiting and Layer already exists messages printed to the console.

This is conceptually similar to what we do with assets, but I'm guessing we'll need to implement it all in the cloud framework JavaScript; for example, check to see if the image was built before building it, and if the digest doesn't change, we can skip the push (provided we know one succeeded before, which may require hitting the registry to ask, which may be more work than it's worth).

[joeduffy]

I was talking to @ellismg just now about https://github.com/pulumi/pulumi-service/issues/979, and Docker build's --cache-from flag. It occurs to me that we may be able to get great Docker caching without needing a build service simply through the intelligent use of --cache-from inside of the cloud abstractions, much in the same way I proposed using the SHA hash above.

The idea is that we'd simply automatically tack on --cache-from to our build arguments when we know of a prior image that we had published for the container in question. Problem solved?

/cc @pgavlin @mmdriley @lukehoban

[joeduffy]

(Note that we have no way of retrieving this information currently inside of a Pulumi program, but I am envisioning some sort of Resource.getOldState() API, which is imminently and easily doable.)

[mmdriley]

Yes, that's my plan for https://github.com/pulumi/pulumi-service/issues/979. @lukehoban and I discussed it and agreed we'd want to prototype it in our own deployment first and tweak the details before we add more complexity to the already-complex Docker build capability here.

[mmdriley]

As opposed to checking old state, we can also do something like pushing the image with an additional tag of repo:yyyymmdd, then using --cache-from targeting that. The --cache-from argument works the way we might hope and just silently ignores the image if it doesn't exist.

This allows us to force an occasional Docker rebuild so that, e.g., users don't end up pinned to old dependency versions.

[joeduffy]

That seems quite reasonable. I do think the cache use policy is something that ought to be kept as "policy" that we can layer atop whatever mechanism we choose here.

The nice thing is I can actually envision us weaning our customer off the current private clouds model in favor of the new managed stacks plus this capability, if it is truly achievable. Given that, we may actually want to pull this in sooner, if it helps us stay sane with respect to supporting legacy clouds.

Pulling back to M14 at least to have a conversation around this. Exciting!

[lukehoban]

@pgavlin @mmdriley Have we addressed this as part of the recent --cache-from changes? If not, what is the specific remaining work we are considering here?

[pgavlin]

I don't believe so, no. I think we're still pushing images even if they haven't changed.

[joeduffy]

We are. Pat had the idea to just check the digest that we get back from the pull when cacheFrom is set. I had assumed we'd just diff old outputs, but I think this would work also when cacheFrom is enabled). Note that this has ABA issues, but that may be okay since in general concurrent pushes are already racey.

[lukehoban]

This issue was moved to pulumi/pulumi-docker#32