config-reloader-sidecar

A small (3MB uncompressed docker image), efficient (via inotify) sidecar to trigger application reloads when configuration changes.

Rationale

A fairly common way to implement configuration hot-reloading is to have the app re-read configuration files when receiving a specific unix signal, usually SIGHUP.

Applications using this method include:

• nginx
• apache
• sshd
• gunicorn
• uwsgi
• mysqld
• logstash
• postgresql
• pgbouncer

In Kubernetes, the "recommended" / usual way of managing configuration is instead to:

• Have a ConfigMap (or Secret) holding the configuration
• When it changes, trigger a rolling-upgrade of the matching Deployment or DaemonSet

While this method is great to ensure configuration changes are highly visible and ensures all replicas use the same config (see immutable infrastructure), it is best for stateless apps which can easily handle rolling upgrades.

Stateful apps, by contrast, might be better off reloading their config so as to not disrupt long-lived open connections, or not to incur a long restart time.

In Kubernetes, an upgrade to a ConfigMap or Secret is eventually (note: see gotchas) propagated to the running Pods, meaning we can watch configuration loaded via ConfigMaps or Secrets and send a signal to the main app when it changes, triggering a hot reload.

The config-reloader-sidecar exists specifically for that use case!

How it works

config-reloader-sidecar uses Go's fsnotify package to watch one (or more) configuration folders, and send a signal to a process when any change is detected within that folder. This includes file created, file updated, file renamed & file deleted, but excludes file permissions changes.

config-reloader-sidecar needs to run, as the name implies, as a separate container in the same Pod as the application you want to reload, i.e. a sidecar.

In addition, you'll need to set shareProcessNamespace: true on your Pod to send signals across containers.

config-reloader-sidecar is then configured through the following env vars:

• CONFIG_DIR: comma-separated list of configuration directories to watch (mandatory)
• PROCESS_NAME: process to send the signal to (mandatory)
• RELOAD_SIGNAL: signal to send (optional, defaults to SIGHUP)
• VERBOSE: whether to print out all inotify events (optional, defaults to false)

Example Pod configuration

apiVersion: v1
kind: Pod
metadata:
  name: auto-reloading-pgbouncer
spec:
  shareProcessNamespace: true
  containers:
  - name: pgbouncer
    image: bitnami/pgbouncer:latest
    command:
    - pgbouncer
    args:
    - /etc/pgbouncer/pgbouncer.ini
    volumeMounts:
    - mountPath: /etc/pgbouncer/
      name: secret-volume
  - name: config-reloader-sidecar
    image: some-private-repo.example.com/config-reloader-sidecar # Note: this isn't yet available on the Docker hub!
    env:
    - name: CONFIG_DIR
      value: /etc/pgbouncer/
    - name: PROCESS_NAME
      value: pgbouncer
    volumeMounts:
    - mountPath: /etc/pgbouncer/
      name: secret-volume
  volumes:
  - name: secret-volume
    secret:
      secretName: pgbouncer-secret

Gotchas

Share Process Namespace

In order for the sidecar to find which process to send the signal to, the Pod needs to be configured to Share Process Namespace with shareProcessNamespace: true.

Update speed

You might noticed when editing a Secret or ConfigMap that your process isn't being reloaded immediately.

This is because the projected values of ConfigMaps and Secrets are not updated exactly when the underlying object changes, but instead they're updated periodically according to the syncFrequency argument to the kubelet config. This defaults to 1 minute.

Config files mounted via subPath are never updated

This is a long-standing Kubernetes issue: ConfigMap and Secrets mounted as files with a subPath key do not get updated by the kubelet. See issue #50345 on Github.

The (pretty ugly) workaround involves mounting the secret/configmap without subPath in a different folder and manually creating a symlink from an initContainer ahead of time to that folder, or if possible at all switching to not using subPath.