docker-compose file for starting multiple CARLA servers

[markcoletti]

We are using multiple CARLA servers running in parallel to train autonomous vehicle models using an evolutionary algorithm (EA). We had previously used a bash script that would spin up a CARLA server for each available GPU. Since this script used the CARLA docker image to spin up the servers, we decided to switch to using a docker-compose file to spin up and down CARLA servers. This not only allowed us to have docker-compose automatically restart failed servers, but to manage the logging such that we can easily look at the logs for a specific CARLA server to diagnose problems.

I'm sharing the docker-compose.yml file that I use to do this; it uses the 0.9.5 image because, unfortunately, that's the latest version of the corresponding client that pip install carla yields.

#
# Developed at the Oak Ridge National Laboratory (ORNL) by Mark Coletti
# ([email protected])
#
# docker-compose up -d
#
version: "3.6"
services:
  carla0:
    restart: unless-stopped
    ports:
      - "2000-2002:2000-2002"
    environment:
      NVIDIA_VISIBLE_DEVICES: 0
      CUDA_VISIBLE_DEVICES: 0
      SDL_VIDEODRIVER: "offscreen"
      SDL_HINT_CUDA_DEVICE: 0
      DISPLAY:
    image: "carlasim/carla:0.9.5"
    command: "/bin/bash ./CarlaUE4.sh -opengl -carla-port=2000 -world-port=2000"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: [gpu]
            device_ids: ["0"]
  carla1:
    restart: unless-stopped
    ports:
      - "3000-3002:3000-3002"
    environment:
      NVIDIA_VISIBLE_DEVICES: 1
      CUDA_VISIBLE_DEVICES: 1
      SDL_VIDEODRIVER: "offscreen"
      SDL_HINT_CUDA_DEVICE: 1
      DISPLAY:
    image: "carlasim/carla:0.9.5"
    command: "/bin/bash ./CarlaUE4.sh -opengl -carla-port=3000 -world-port=3000"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: [gpu]
            device_ids: ["1"]
  carla2:
    restart: unless-stopped
    ports:
      - "4000-4002:4000-4002"
    environment:
      NVIDIA_VISIBLE_DEVICES: 2
      CUDA_VISIBLE_DEVICES: 2
      SDL_VIDEODRIVER: "offscreen"
      SDL_HINT_CUDA_DEVICE: 2
      DISPLAY:
    image: "carlasim/carla:0.9.5"
    command: "/bin/bash ./CarlaUE4.sh -opengl -carla-port=4000 -world-port=4000"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: [gpu]
            device_ids: ["2"]
  carla3:
    restart: unless-stopped
    ports:
      - "5000-5002:5000-5002"
    environment:
      NVIDIA_VISIBLE_DEVICES: 3
      CUDA_VISIBLE_DEVICES: 3
      SDL_VIDEODRIVER: "offscreen"
      SDL_HINT_CUDA_DEVICE: 3
      DISPLAY:
    image: "carlasim/carla:0.9.5"
    command: "/bin/bash ./CarlaUE4.sh -opengl -carla-port=5000 -world-port=5000"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: [gpu]
            device_ids: ["3"]

In case you are not familiar with docker-compose, all of the following must be done for the same directory that contains the above docker-compose.yml file.

• To spin up the servers: docker-compose up -d --remove-orphans
• To spin them down: docker-compose down
• To check if servers are running: docker-compose ps
• To show last 30 lines of carla2's logs: dco logs --tail=30 carla2

There may be a more elegant implementation where a single server can be specified, and the use of docker-compose's --scale parameter can be used to freely dial up and down the desired number of coincident CARLA servers. But, for now, this docker-compose.yml meets our needs, and hopefully it may be useful to someone else.