Docker Running multiple daemons

This is a list of details on how to run multiple docker daemons on a single host under gentoo.
I found this useful from a point of view of running one daemon for development (default) and the other for production (-prod).

The idea being to avoid deleting anything inside the prod setup when messing with containers or images inside the default development daemon.
So two separate isolated docker daemons, one that might contain a wiki or gitea server and a second for development / messing around with.

• https://www.jujens.eu/posts/en/2018/Feb/25/multiple-docker/
• https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

Docker Config Setup

First we want to setup a second init script for docker-prod (production)

cd /etc/init.d/
ln -s /etc/init.d/docker /etc/init.d/docker-prod

Next we want to copy over the conf.d file

cd /etc/conf.d/
cp docker docker-prod

conf.d files

For /etc/conf.d/docker we should be able to leave this as is assuming it's already been setup to look at /etc/docker/daemon.json

Next let's edit /etc/conf.d/docker-prod

# /etc/conf.d/docker-prod: config file for /etc/init.d/docker-prod
DOCKER_LOGFILE="/var/log/docker-prod.log"
DOCKER_PIDFILE="/run/docker-prod.pid"
DOCKER_OPTS="--host unix:///var/run/docker-prod.sock --exec-root /var/run/docker-prod --config-file /etc/docker/daemon-prod.json"

For this second docker daemon we are changing a few settings so that they don't overlap with the default docker

• log file -> /var/log/docker-prod.log
• pid file -> /run/docker-prod.pid
  used to track the process id
• data root -> /mnt/vol2/var/docker-prod
  used for storing containers / images
• host socket -> unix:///var/run/docker-prod.sock
  this allows us to connect to it from the cli using a different socket
• exec root -> /var/run/docker-prod
  this is where docker stores it's temporary exec files
• config file -> /etc/docker/daemon-prod.json
  used for additional settings

/etc/docker/ daemon config files

Next lets create a new /etc/docker/daemon-prod.json file
This contains a different network default subnet range for creating new networks than the first docker daemon.

We also set bridge to none for the default bridge.
This avoids any conflicts with two docker daemons both trying to use the docker0 name for the bridge at the same time.
Also I've discovered it's better to use a user defined network anyway for other reasons.

{
 "data-root": "/mnt/vol2/var/docker-prod",
 "bridge": "none",
 "default-address-pools":
 [
  {"base":"10.101.0.0/16","size":24}
 ]
}

Start docker / Setup Portainer

Next lets start the new docker instance up

# Startup
/etc/init.d/docker-prod start

# Add to default run level
rc-update add docker-prod default

For docker-prod we now just need to use the following option whenever we want to connect to it from the cli

-H unix:///var/run/docker-prod.sock

Lets create a new default network for it

docker network create --driver=bridge --subnet=10.101.0.0/24 --gateway=10.101.0.1 defnet-prod \
-o "com.docker.network.bridge.enable_ip_masquerade"="true" \
-o "com.docker.network.bridge.enable_icc"="true" \
-o "com.docker.network.bridge.host_binding_ipv4"="0.0.0.0" \
-o "com.docker.network.driver.mtu"="1500"

So to setup portainer on it

docker -H unix:///var/run/docker-prod.sock volume create portainer_data
docker -H unix:///var/run/docker-prod.sock run -d -p 8001:8000 -p 9001:9000 \
  --name=portainer --restart=always --network=defnet-prod \
  -v /var/run/docker-prod.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

Note

• The use of a different port number 80001 / 9001 to avoid conflicting with the portainer on the default docker.
  -p 8001:8000 -p 9001:9000

• The use of a different socket so that it connects to the correct docker instance
  -v /var/run/docker-prod.sock:/var/run/docker.sock