Core 884.0.0 - Enabling SELinux in prevents Docker containers from running

[tjdett]

When starting a new instance with this cloud-config:

#cloud-config
coreos:
  update:
    reboot-strategy: "off"
  units:
  - name: setup-selinux.service
    command: start
    enable: true
    content: |
      [Unit]
      Description=Enable SELinux
      ConditionPathExists=!/var/lib/selinux-has-been-setup
      Before=early-docker.service
      [Service]
      Type=oneshot
      RemainAfterExit=yes
      ExecStart=/opt/bin/setup-selinux
      ExecStartPost=/usr/bin/touch /var/lib/selinux-has-been-setup
write_files:
- path: /opt/bin/setup-selinux
  permissions: 0744
  owner: root
  content: |
    #!/usr/bin/env bash
    set -ex
    rm /etc/audit/rules.d/80-selinux.rules
    rm /etc/audit/rules.d/99-default.rules
    rm /etc/selinux/mcs
    cp -a /usr/lib/selinux/mcs /etc/selinux
    rm /var/lib/selinux
    cp -a /usr/lib/selinux/policy /var/lib/selinux
    semodule -DB
    systemctl restart audit-rules
    cp --remove-destination $(readlink -f /etc/selinux/config) /etc/selinux/config
    sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
    setenforce 1
manage_etc_hosts: localhost

Running a simple Docker container fails:

$ docker run -ti --rm busybox        
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
039b63dd2cba: Pull complete 
c51f86c28340: Pull complete 
Digest: sha256:eb3c0d4680f9213ee5f348ea6d39489a1f85a318a2ae09e012c426f78252a6d2
Status: Downloaded newer image for busybox:latest
Error response from daemon: Cannot start container 981a3ab0ba9925608a3e083bca37691e2b8154a8adb7412afbebcf051abebc2c: [8] System error: permission denied

And produces the following AVC messages:

audit[922]: AVC avc:  denied  { entrypoint } for  pid=922 comm="exe" path="/bin/sh" dev="overlay" ino=17490 scontext=system_u:system_r:svirt_lxc_net_t:s0:c551,c767 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
kernel: audit: type=1400 audit(1449218666.592:154): avc:  denied  { entrypoint } for  pid=922 comm="exe" path="/bin/sh" dev="overlay" ino=17490 scontext=system_u:system_r:svirt_lxc_net_t:s0:c551,c767 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="resolv.conf" dev="vda9" ino=290 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="hostname" dev="vda9" ino=287 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="hosts" dev="vda9" ino=286 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="resolv.conf" dev="vda9" ino=290 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="hostname" dev="vda9" ino=287 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0
audit[828]: AVC avc:  denied  { relabelfrom } for  pid=828 comm="docker" name="hosts" dev="vda9" ino=286 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:svirt_lxc_file_t:s0:c551,c767 tclass=file permissive=0

[kayrus]

@mjg59 looks like we don't provide selinux module for docker. did I miss something or not? I tried to compile own docker.pp and semodule -i docker.pp, but at first glance it doesn't help. https://github.com/kayrus/docker-selinux

relates to coreos/coreos-overlay#1632 , moby/moby#15498 , moby/moby#15832, sallyom/docker@04329e0, coreos/docker@37fb0ab and probably this "Remove LXC" commit moby/moby@3b5fac4
Docker 1.8.3 works well with enabled selinux.

[mjg59]

@kayrus Docker is using the same selinux policy as rkt, and it's provided via the lxc policy

[kayrus]

@mjg59 Probably Docker <= 1.8.3 did. Why does new 1.9.1 fail?

[mjg59]

Docker isn't labelling the overlay directories. I've got a slightly hacky patch to do this, I'll clean it up tomorrow.

[mjg59]

coreos/docker#10 coreos/coreos-overlay#1703 and coreos/linux#5

[mjg59]

Should be fixed in the next Alpha. This turned out to be a combination of problems:

1. The kernel was mishandling selinux checks when creating new files on overlayfs mounts
2. Docker requires a slightly different selinux policy
3. Docker wasn't labeling the underlying overlayfs mountpoints

The reason this worked in previous releases is that Docker was being built without selinux support, so wasn't transitioning to a confined context on container startup.

[marineam]

FYI, 921.0.0 will include the kernel and policy fixes but by mistake we missed the labeling fix in coreos/docker#10 but that will come soon.

[crawford]

This was fixed in 926.0.0.