Pen and Systemd

Systemd is an init system for Linux, i.e. a program which runs as PID 1 and controls the startup of daemons and services. It does a bunch of other stuff as well, in a way that isn’t quite in keeping with Unix tradition, and this has caused a bit of controversy. We can ignore that for the purpose of this post.

Red Hat 7 uses systemd as its default init, as will Debian 8. Systemd isn’t configured like the familiar SysV init, so most people tasked with installing Linux servers will need to relearn. For this post, we will look at installing and configuring Pen on a CentOS 7 server.

First we need the Pen binaries. Fortunately that job has already been done for us. Pen is in the “Extra Packages for Enterprise Linux” repository, or EPEL:

yum install epel-release
yum --enablerepo=epel -y install pen

Create a user for pen to run as:

useradd pen

Create a directory for pen to keep its stuff while it is running. We can’t use /var/run because the pen user isn’t allowed to create files there, and we can’t just mkdir /var/run/pen because /var/run is a tmpfs which is recreated when the server boots. Instead we create this file in /etc/tmpfiles.d/:

# /etc/tmpfiles.d/pen.conf
d /var/run/pen 0755 pen pen -

And to actually create the directory:

systemd-tmpfiles --create

Create the configuration files, one per load balanced service. In this case, one for dns and one for http.

[root@centos7 pen]# cat /etc/pen/www.cfg
http
server 0 address 194.9.95.65 port 80
debug 1
[root@centos7 pen]# cat /etc/pen/dns.cfg
server 0 address 8.8.8.8 port 53
server 1 address 8.8.4.4 port 53
debug 1

We have debugging turned on in order to get some logging. In order to get rsyslog to handle the debug messages, add this line to /etc/rsyslog.conf:

*.debug /var/log/debug

Restart rsyslog like so:

systemctl restart rsyslog

Finally add the service files that tell systemd to manage Pen.

[root@centos7 pen]# cat /usr/lib/systemd/system/pen-www.service
[Unit]
Description=Pen load balancer (www)
[Service]
Type=forking
PIDFile=/var/run/pen/www.pid
ExecStart=/usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80
[root@centos7 pen]# cat /usr/lib/systemd/system/pen-dns.service
[Unit]
Description=pen load balancer (dns)
[Service]
Type=forking
PIDFile=/var/run/pen/dns.pid
ExecStart=/usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53

Make systemd reread its configuration. That can actually be done in the oldfashioned way:

kill -HUP 1

Now we can use systemctl to start and stop the services:

[root@centos7 pen]# systemctl start pen-www
[root@centos7 pen]# systemctl start pen-dns
[root@centos7 pen]# systemctl status pen-dns
pen-dns.service - pen load balancer (dns)
Loaded: loaded (/usr/lib/systemd/system/pen-dns.service; static)
Active: active (running) since Sun 2014-10-19 02:53:36 CEST; 38s ago
Process: 19004 ExecStart=/usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53 (code=exited, status=0/SUCCESS)
Main PID: 19005 (pen)
CGroup: /system.slice/pen-dns.service
└─19005 /usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53

Oct 19 02:53:36 centos7 systemd[1]: Starting pen load balancer (dns)...
Oct 19 02:53:36 centos7 systemd[1]: PID file /var/run/pen/dns.pid not readable (yet?) after start.
Oct 19 02:53:36 centos7 systemd[1]: Started pen load balancer (dns).
[root@centos7 pen]# systemctl status pen-www
pen-www.service - Pen load balancer (www)
Loaded: loaded (/usr/lib/systemd/system/pen-www.service; static)
Active: active (running) since Sun 2014-10-19 02:53:33 CEST; 47s ago
Process: 19000 ExecStart=/usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80 (code=exited, status=0/SUCCESS)
Main PID: 19001 (pen)
CGroup: /system.slice/pen-www.service
└─19001 /usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80

Oct 19 02:53:33 centos7 systemd[1]: Starting Pen load balancer (www)...
Oct 19 02:53:33 centos7 systemd[1]: PID file /var/run/pen/www.pid not readable (yet?) after start.
Oct 19 02:53:33 centos7 systemd[1]: Started Pen load balancer (www).

Let’s check that it works, too.

[root@centos7 pen]# dig +short @localhost siag.nu
194.9.95.65

What a relief. :)

Thanks to Djamel Ouerdi for helping me figure out how this works.