Redis sentinel is not working as expected for dispatcher service

[dot-mike]

Support guidelines

• I've read the support guidelines

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I have a working set-up using redis sentinel. I wanted to implement distributed poller set-up with redis using sentinel as the locking mechanism. Implementing redis-sentinel was not easy, but getting it to work with librenms was even harder as the docs is lacking.

I had some issues with dispatcher container not connecting to my redis-sentinel host and it it turns out having both the environment variable REDIS_HOST & REDIS_SENTINEL variable will cause issues.

So, the main issue as it is today is that both the dispatcher service (python) and librenms (php) requires redis backend. While the dispatcher service supports redis sentinel for discovering redis-master, the librenms application does not. See issue librenms/librenms#16792

The default for all docker services is to include the librenms.env as an env_file effectively setting all the variables defined in the file as environment variables inside the service. While this is handy, this will also cause issues when you want to use the REDIS_SENTINEL environment variable. The issue is that the env-variable REDIS_HOST is preferred over REDIS_SENTINEL as it happens in this IF-case here:

docker/rootfs/etc/cont-init.d/05-svc-dispatcher.sh

Line 97 in 1d7c972

elif [ -n "$REDIS_HOST" ]; then

So if the user sets the environment-variables as specified in the docs for distributed poller configuration inside the .env:

REDIS_SENTINEL=10.0.0.10:26379,10.0.0.11:26379
REDIS_SENTINEL_SERVICE=librenms-master
REDIS_SENTINEL_PASSWORD=mypassword

and then include the variables in the compose.yml file:

....
    environment:
      ....
      - "REDIS_SENTINEL=${REDIS_SENTINEL}"
      - "REDIS_SENTINEL_SERVICE=${REDIS_SENTINEL_SERVICE}"
      - "REDIS_SENTINEL_PASSWORD=${REDIS_SENTINEL_PASSWORD}"

it will simply not work and you will see errors in the logs

docker compose logs

librenms_dispatcher  | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
librenms_dispatcher  | [s6-init] ensuring user provided files have correct perms...exited 0.
librenms_dispatcher  | [fix-attrs.d] applying ownership & permissions fixes...
librenms_dispatcher  | [fix-attrs.d] done.
librenms_dispatcher  | [cont-init.d] executing container initialization scripts...
librenms_dispatcher  | [cont-init.d] 00-fix-logs.sh: executing...
librenms_dispatcher  | [cont-init.d] 00-fix-logs.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 01-fix-uidgid.sh: executing...
librenms_dispatcher  | [cont-init.d] 01-fix-uidgid.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 02-fix-perms.sh: executing...
librenms_dispatcher  | Fixing perms...
librenms_dispatcher  | [cont-init.d] 02-fix-perms.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 03-config.sh: executing...
librenms_dispatcher  | Setting timezone to Europe/Berlin...
librenms_dispatcher  | Setting PHP-FPM configuration...
librenms_dispatcher  | Setting PHP INI configuration...
librenms_dispatcher  | Setting OpCache configuration...
librenms_dispatcher  | Setting Nginx configuration...
librenms_dispatcher  | Updating SNMP community...
librenms_dispatcher  | Initializing LibreNMS files / folders...
librenms_dispatcher  | Setting LibreNMS configuration...
librenms_dispatcher  | Checking LibreNMS plugins...
librenms_dispatcher  | Fixing perms...
librenms_dispatcher  | Checking additional Monitoring plugins...
librenms_dispatcher  | Checking alert templates...
librenms_dispatcher  | [cont-init.d] 03-config.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 04-svc-main.sh: executing...
librenms_dispatcher  | [cont-init.d] 04-svc-main.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 05-svc-dispatcher.sh: executing...
librenms_dispatcher  | >>
librenms_dispatcher  | >> Sidecar dispatcher container detected
librenms_dispatcher  | >>
librenms_dispatcher  | Waiting 60s for database to be ready...
librenms_dispatcher  | Database ready!
librenms_dispatcher  | NODE_ID: dispatcher1
librenms_dispatcher  | Setting Redis
librenms_dispatcher  | [cont-init.d] 05-svc-dispatcher.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 06-svc-syslogng.sh: executing...
librenms_dispatcher  | [cont-init.d] 06-svc-syslogng.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 07-svc-cron.sh: executing...
librenms_dispatcher  | [cont-init.d] 07-svc-cron.sh: exited 0.
librenms_dispatcher  | [cont-init.d] 08-svc-snmptrapd.sh: executing...
librenms_dispatcher  | [cont-init.d] 08-svc-snmptrapd.sh: exited 0.
librenms_dispatcher  | [cont-init.d] ~-socklog: executing...
librenms_dispatcher  | [cont-init.d] ~-socklog: exited 0.
librenms_dispatcher  | [cont-init.d] done.
librenms_dispatcher  | [services.d] starting services
librenms_dispatcher  | [services.d] done.
librenms_dispatcher  | dispatcher1(CRITICAL):ERROR: Redis connection required for distributed polling
librenms_dispatcher  | dispatcher1(CRITICAL):Lock manager could not connect to Redis. AuthenticationError: Authentication required.

The log-line Setting Redis is the hint we need that tells us sentinel is not being used for locking mechanism! So, instead we can modify the compose.yml and override REDIS_HOST by setting it manually in environment-section to an empty value:

  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: librenms-dispatcher
    ....
    environment:
      - "REDIS_HOST="

So now the dispatcher service can connect to the sentinel host to get true master :)

If you so happens to have a password protected redis-host for the master and replica nodes (which you should BTW!), the password stored in REDIS_SENTINEL_PASSWORD will not used to connect to the true replica master.
The password specified in the varaible REDIS_SENTINEL_PASSWORD will only be used for connecting to the sentinel hosts to get true master. It will not be used for the actual redis hosts. It so happens that you can have different passwords for sentinel and actual redis hosts! For the dispatcher to connect to the redis master it also needs to password which is set in environment variable REDIS_PASSWORD.

In the end this is what my dispatcher-service config looks like in compose:

  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: mydispatcher123
    cap_add:
      - NET_ADMIN
      - NET_RAW
    depends_on:
      - librenms
    volumes:
      - "./librenms:/data"
    env_file:
      - "./librenms.env"
    environment:
      - "APP_KEY=${APP_KEY}"
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "REDIS_HOST="
      - "REDIS_PASSWORD=mypassword"
      - "DB_HOST=10.0.0.10"
      - "DB_CONNECTION=mysql_cluster"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "DISPATCHER_NODE_ID=mydispatcher123"
      - "SIDECAR_DISPATCHER=1"
      - "REDIS_SENTINEL=${REDIS_SENTINEL}"
      - "REDIS_SENTINEL_SERVICE=${REDIS_SENTINEL_SERVICE}"
      - "REDIS_SENTINEL_PASSWORD=${REDIS_SENTINEL_PASSWORD}"
    restart: always

Expected behaviour

.

Actual behaviour

.

Steps to reproduce

.

Docker info

.

Docker Compose config

.

Logs

.

Additional info

.