[Services] Restart DHCP-Relay service upon unexpected critical process exit.

[yozhao101]

• What I did
  Restart DHCP-Relay service if one of critical processes running in DHCP-Relay container exited or crashed abnormally.

• How I did it
  Generally I follow the framework created by Joe to implement this feature in DHCP-Relay container.
  First, add supervisor-proc-exit-listener event listener option in Supervisord configuration file in DHCP-Relay docker container. Supervisord will read a list of critical processes for which to monitor the unexpected crashed and exited.
  For DHCP-Relay container, since a bunch of critical processes will be monitored by a group, we only need put the groupname in the file critical_processes. At the same time, we also add source code in supervisor-proc-exit-listener script to retrieve the groupname and then decide whether it appears in critical_processes.
  Second, configure dhcp-relay.service to always auto-restart the service if it stops, with a delay of 30 seconds. Also set a rate limit of 3 restarts within 20 minutes (1200 seconds).

• How to verify it
  On your switch device, please use docker ps command to list all running docker containers.
  Then use docker exec -it container_id /bin/bash to login target container. Typing top command
  on the shell will display all the processes dynamically and you will spot the process id of one
  of the critical processes. Finally type the command kill -9 process_id to terminate one process.
  After exiting the container, you can use watch -n 1 docker ps to dynamically see the restart
  of DHCP-Relay container.

[jleveque]

Looks good. However, I feel that it's not clear that one can now add group names to the "critical_processes" file (because the file name doesn't mention groups). I don't want to rename the file to something long, like "critical_processes_and_groups", though. Any suggestions?

[yozhao101]

I though this issue for a while. In order to keep consistency with other containers, we can put
actual process names in this file not the group name. Or we can divide the critical_processes
into two sections: critical process section and group section. For dhcp-relay, we can leave the critical
process section empty and just put a single group name in group section. For now, can we create a critical_processes.j2 file to handle this issue?

[jleveque]

I don’t think we should take the templated approach, as using the group name is now shown to work, is much simpler and will require far less maintenance in the future. I think we can keep this as-is for now, but I would like to distinguish between processes and groups in the future. Maybe once all of the containers are managed properly, we can update the critical_processes syntax to match the supervisor.conf syntax. E.g.,

program:x
program:y
group:z

Then we can update the event listener's parsing logic. This separates the individual processes from the groups and also makes it clear to the user.

[jleveque]

Retest this please

[jleveque]

Retest vs please