Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker_container devices forces resource replacement at all times #603

Open
acolpan opened this issue Feb 3, 2024 · 12 comments
Open

docker_container devices forces resource replacement at all times #603

acolpan opened this issue Feb 3, 2024 · 12 comments

Comments

@acolpan
Copy link

acolpan commented Feb 3, 2024

Terraform v1.7.2
kreuzwerker/docker v3.0.2

To reproduce this issue:

  1. Locally, provision a docker_container resource using the nested schema for devices (i.e. as below)
devices {
  host_path = "dev/net/tun"
}
  1. Create the resource (i.e. terraform apply --auto-approve)

  2. Right after the step 2, after a container resource is provisioned, please note that, at this moment, the state is not changed anywhere because it's just been provisioned, try the following command to see the state "# forces replacement" further below.

  terraform plan

  - devices { # forces replacement
      - container_path = "dev/net/tun" -> null
      - host_path      = "dev/net/tun" -> null
      - permissions    = "rwm" -> null
    }
  + devices { # forces replacement
      + host_path = "dev/net/tun"
    }

This should not happen! The "terraform plan" should come out as clean in full agreement with the current state of the docker container. Though the devices after the resource is provisioned works as expected, it does not pass the "terraform plan" state check. This probably is an issue with the kreuzwerker/docker provider where it misses the container state for the devices when it inquires the state of the container.

A sample configuration to reproduce this issue: main.tf

Please replace the PATH_TO_CONFIG_DIR and the PATH_TO_DOWNLOADS_DIR with anything that works for you.

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 3.0.2"
    }
  }
}

provider "docker" {
  host = "unix:///var/run/docker.sock"
}

data "docker_registry_image" "qbittorrent" {
  name = "linuxserver/qbittorrent:latest"
}

resource "docker_image" "qbittorrent" {
  name          = data.docker_registry_image.qbittorrent.name
  pull_triggers = [data.docker_registry_image.qbittorrent.sha256_digest]
}

resource "docker_container" "qbittorrent" {
  name = "qbittorrent"
  capabilities {
    add = ["NET_ADMIN"]
  }

  devices {
    host_path = "dev/net/tun"
  }

  ports {
    internal = "6881"
    external = "6881"
  }
  ports {
    internal = "8090"
    external = "8090"
  }
  env = ["PUID=1000", "PGID=1000", "TZ=America/Chicago", "UMASK_SET=022", "WEBUI_PORT=8090"]
  volumes {
    host_path      = "PATH_TO_CONFIG_DIR/config"
    container_path = "/config"
  }
  volumes {
    host_path      = "PATH_TO_DOWNLOADS_DIR/downloads"
    container_path = "/downloads"
  }
  restart = "always"
  image   = docker_image.qbittorrent.image_id
}

Steps to reproduce the issue with the above configuration:

  1. Run the following command to provision the resources
terraform apply --auto-approve
  1. Verify that both the qbittorrent image as well as the container are successfully created, and the container is up and running

  2. Run the following command to verify that the state of the qbittorrent container also looks all good including the devices

terraform state show docker_container.qbittorrent
  1. Run the following command to see the state mismatch due to the devices
terraform plan

If you remove the devices, then the states, terraform's vs docker's, agree but the container does not work as expected due to the missing "dev/net/tun" set by the devices. If you keep the devices, then the container works as expected but the states do not agree, and the terraform wants to replace the container.

@acolpan acolpan changed the title Nested schema for devices in resource (docker_container) forces resource replacement without a state change. Nested schema for devices in resource (docker_container) forces resource replacement even without a state change. Feb 3, 2024
@acolpan acolpan changed the title Nested schema for devices in resource (docker_container) forces resource replacement even without a state change. devices in resource (docker_container) forces resource replacement even without a state change. Feb 3, 2024
@acolpan acolpan changed the title devices in resource (docker_container) forces resource replacement even without a state change. docker_container devices forces resource replacement even without a state change Feb 4, 2024
@acolpan acolpan changed the title docker_container devices forces resource replacement even without a state change docker_container devices forces resource replacement at all times Feb 5, 2024
@IlyesDemineExtVeolia
Copy link

@mavogel

@tloesch
Copy link

tloesch commented Apr 26, 2024

@acolpan
We encountered similar problems today. Numerous Docker provider resources are consistently being replaced, even when a replacement shouldn't be necessary. After some debugging, we found that the Docker version on our server had been updated to version 26+. Once we downgraded it to Version 25.0.3, the provider started behaving as expected. Upon reviewing the Docker changelogs, we noticed that changes were made to their API, which could potentially lead to these issues. Unfortunately, this provider is still unmaintained, and a solution is unlikely to be published here anytime soon. You might want to explore some forks of this provider, as the issue might be resolved there. Alternatively, consider switching to a different tool for your Docker-related tasks.

It would be appreciated if you could confirm whether downgrading Docker on the server resolved the issue in your situation.

@enc
Copy link
Contributor

enc commented Apr 26, 2024

Hi,
we will be looking into this project again. More on that in the next weeks.

@acolpan
Copy link
Author

acolpan commented Apr 27, 2024

@acolpan We encountered similar problems today. Numerous Docker provider resources are consistently being replaced, even when a replacement shouldn't be necessary. After some debugging, we found that the Docker version on our server had been updated to version 26+. Once we downgraded it to Version 25.0.3, the provider started behaving as expected. Upon reviewing the Docker changelogs, we noticed that changes were made to their API, which could potentially lead to these issues. Unfortunately, this provider is still unmaintained, and a solution is unlikely to be published here anytime soon. You might want to explore some forks of this provider, as the issue might be resolved there. Alternatively, consider switching to a different tool for your Docker-related tasks.

It would be appreciated if you could confirm whether downgrading Docker on the server resolved the issue in your situation.

After upgrading to docker version 26.1.0, aside from the devices property I was having this issue with, I started experiencing a similar issue with another property named network_mode. However, there's a workaround to this problem. You can use a lifecycle property and ignore changes to those properties as in the following resource example I am sending you. I hope this helps.

resource "docker_container" "qbittorrent" {
 .
 .
 .
  lifecycle {
    ignore_changes = [
      devices,
      network_mode,
    ]
  }
  }

@acolpan
Copy link
Author

acolpan commented Apr 27, 2024

Hi, we will be looking into this project again. More on that in the next weeks.

Thanks. As I responded to @tloesch, there's a workaround to this issue. However, ignoring the changes to those properties is limiting so a permanent solution is preffered.

@IlyesDemineExtVeolia
Copy link

IlyesDemineExtVeolia commented Apr 29, 2024

@tloesch @acolpan @enc I have the same issue. I use this module (https://github.com/terraform-aws-modules/terraform-aws-lambda) to build and deploy lambdas docker. This module use this provider to build the docker image. The ressource "docker_image" is always rebuild even if context or Dockerfile doesn't changes. 😔
Similar ticket here #607

@tloesch
Copy link

tloesch commented Apr 29, 2024

@tloesch @acolpan @enc I have the same issue. I use this module (https://github.com/terraform-aws-modules/terraform-aws-lambda) to build and deploy lambdas docker. This module use this provider to build the docker image. The ressource "docker_image" is always rebuild even if context or Dockerfile doesn't changes. 😔

@IlyesDemineExtVeolia
I know this not a Long Term Solution. But you can try downgrading docker to Version 25.0.x on the target system.
This Version still gets Security Updates.

@IlyesDemineExtVeolia
Copy link

@tloesch Thanks for your response. I still have the issue with docker 25. I will wait for a permanent fix

@kukukk
Copy link

kukukk commented May 7, 2024

Hi,
Any update on this issue?

@IlyesDemineExtVeolia
Copy link

@enc
Hi,
any news ?

@cdn-bborucki
Copy link

Facing this issue as well.

I poked around in Docker to see what might have broken this and believe this to be the culprit change (at least for replacements due to network_mode): moby/moby@4eed3dc.

What's more is it looks like there's been further refinement to how they specify the default network_mode: moby/moby#48008. Examining this change makes it pretty clear what the new defaults are:

Where the actual strings are defined here: https://github.com/moby/moby/blob/2cfc2a57a84900db112bd911143de7aed719d739/api/types/network/network.go#L9-L20 (spoiler: "bridge" on Unix, "nat" on Windows).

Here's hoping this may help inform a permanent solution!

@cowwoc
Copy link

cowwoc commented Nov 11, 2024

Sounds like a Docker bug to me, because their documentation explicitly states that the default is "bridge": https://docs.docker.com/engine/network/drivers/

Either the documentation is wrong or the implementation is wrong. I filed this bug report: docker/for-win#14417

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants