pulumi refresh does not respect docker:host configuration setting for ssh connection

[moritzheiber]

Describe what happened

According to the documention one can set the DOCKER_HOST variable through a pulumi setting utilizing a ssh connection, i.e.

$ pulumi config set docker:host ssh://[email protected]:22/

However, although commands like update or destroy do respect the configuration setting (i.e. DOCKER_HOST is set properly and the command connects to the right docker daemon via ssh successfully), running a pulumi refresh without passing DOCKER_HOST as a variable will fail with the following error:

Note: the Volume in this example is just one of many resources that fail.

$ pulumi refresh
[...]
Diagnostics:
  docker:index:Volume (volume):
    error: Preview failed: refreshing urn:pulumi:project::environment::docker:index/volume:Volume::volume: 1 error occurred:
    	* Unable to inspect volume: Error response from daemon: get volume: no such volume
[...]

Perusing the logs reveals that pulumi is using the local docker daemon via a socket connection instead of the configuration setting:

I0616 11:03:01.165596  114555 eventsink.go:59] 2024/06/16 11:03:01 successful connection to docker daemon at unix:///var/run/docker.sock
I0616 11:03:01.165821  114555 eventsink.go:59] 2024/06/16 11:03:01 Setting DOCKER_HOST to unix:///var/run/docker.sock

Sample program

import * as pulumi from "@pulumi/pulumi";
import * as docker from "@pulumi/docker";

const volume = new docker.Volume("volume", {
    driver: "local",
    name: "volume",
})

Log output

I can provide further logs upon request. Currently it would require significant work to redact output I cannot submit publicly, and the bug is 100% reproducible with the example I posted above.

Affected Resource(s)

The resources I've tested this with were Volume, Container and Network but I reckon all resources are affected here given the nature of this bug.

Output of pulumi about

$ pulumi about

CLI          
Version      3.119.0
Go Version   go1.22.3
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  docker  4.5.4
language  nodejs  unknown

Host     
OS       ubuntu
Version  24.04
Arch     x86_64

This project is written in nodejs: executable='/nix/store/zyds4d0j2d9yi77kgzzzr2pvfg8idk2m-nodejs-22.2.0/bin/node' version='v22.2.0'

Current Stack: organization/project/project

TYPE                                  URN
pulumi:pulumi:Stack                   urn:pulumi:project::project::pulumi:pulumi:Stack::project
pulumi:providers:docker               urn:pulumi:project::project::pulumi:providers:docker::default_4_5_3
docker:index/volume:Volume            urn:pulumi:project::project::docker:index/volume:Volume::service_data
docker:index/volume:Volume            urn:pulumi:project::project::docker:index/volume:Volume::volume
docker:index/network:Network          urn:pulumi:project::project::docker:index/network:Network::mail
docker:index/network:Network          urn:pulumi:project::project::docker:index/network:Network::main
docker:index/remoteImage:RemoteImage  urn:pulumi:project::project::docker:index/remoteImage:RemoteImage::service
docker:index/container:Container      urn:pulumi:project::project::docker:index/container:Container::service


Found no pending operations associated with project

Backend        
Name           <redacted>
URL            file://~
User           <redacted>
Organizations  
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/docker  4.5.4
@pulumi/pulumi  3.120.0
@types/mocha    10.0.6
@types/node     20.14.2
mocha           10.4.0
ts-node         10.9.2

Pulumi locates its logs in /run/user/1000 by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[moritzheiber]

#710 might be related here

[blampe]

#1106 is a similar issue related to refresh and host values, in case that's relevant here.

Perusing the logs reveals that pulumi is using the local docker daemon via a socket connection instead of the configuration setting:

I think this is actually a red herring. These logs come from the "native" portion of the provider which is only responsible for building images. The Volume resource is managed by the "bridged" portion, which is configured according to upstream/internal/provider/config.go with some patches applied.

running a pulumi refresh without passing DOCKER_HOST as a variable will fail with the following error:

There's a good comment from Pat in pulumi/esc#199 (comment) that explains how refresh determines which configuration to use. In particular, it should be using the previously persisted value of the host.

the bug is 100% reproducible with the example I posted above.

Thank you for including the code sample! Unfortunately I haven't been able to reproduce the issue using local sockets or local tcp listeners (which AFAIK should trigger the same ssh code path).

@moritzheiber can you confirm the order of operations?

1. pulumi config set docker:host ssh://...
2. pulumi up
3. pulumi refresh