Evaluate different build tools for lambda container images

[kwiatekus]

Description

So far we have learned that Kaniko must run as a privileged container - which is a blocker for 1.19

The most important criteria is that it should build without allowance of privileged mode.

• makisu
• img on tob of buildkit buulds as unprivileged user
• hints from google

Attachments
Related issues

• Adapt serverless to 'allowPrivilegedContainers: false' policy #9785
• Introduce proper PodSecurityPolicy for function build job #9779

[pPrecel]

The output of my research:

Introduction:
We'd like to replace kaniko which is used to build docker images inside the Kubernetes container with another tool that can be run in the container without any privileges (like root or custom security profile).

Kaniko as a benchmark:
I used kaniko to compare it with other tools that I described in this comment below. Kaniko is a tool that needs to be in the container run as root and this is disqualifying it

More info:
Image size: 78Mb
Built time: 3m38s

Input yamls:

apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:latest
        imagePullPolicy: IfNotPresent
        args:
        - --context
        - dir:///kaniko-context/..data
        - --dockerfile
        - /kaniko-context/Dockerfile
        - --destination
        - wookieeteam/k3s-kaniko-test:1
        env:
        volumeMounts:
        - name: context
          mountPath: /kaniko-context
        - name: registry-secret
          mountPath: kaniko/.docker/
      volumes:
      - name: context
        configMap:
          name: source
          items:
            - key: requirements.txt
              path: requirements.txt
            - key: main.py
              path: main.py
            - key: Dockerfile
              path: Dockerfile
      - name: registry-secret
        secret:
          secretName: docker-registry-kaniko
---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-kaniko
stringData:
  config.json: '{"auths": {"https://index.docker.io/v1/": {"auth": "xxxxxxxxxxxxxxxxxx"}}}'
type: Opaque

I found some tools and next problems:

• buildkit/Img which are really nice tools to build images inside the Kubernetes containers (Img is based on the buildkit) but those tools need be run in the container with custom security profile and I think this is not resolving our problems because of security reasons (Am I right?). The project provides nice documentation of the project so applying it instead of the kanino should be easy

Few more data:
Image size: 134Mb for buildkit and 49Mb for Img
Built time: 4m21s

Input yamls:

apiVersion: batch/v1
kind: Job
metadata:
  name: img
spec:
  template:
    metadata:
      annotations:
        # gardener will reject creation of pods with custom security profiles 
        container.apparmor.security.beta.kubernetes.io/img: unconfined
        container.seccomp.security.alpha.kubernetes.io/img: unconfined
    spec:
      restartPolicy: Never        
      containers:
        - name: img
          image: r.j3ss.co/img
          env:
            - name: DOCKER_CONFIG
              value: /usr/.docker
          args:
            - build
            - --output
            - type=image,name=docker.io/wookieeteam/k3s-img-test:1,push=true
            - -t 
            - wookieeteam/k3s-img-test:1
            - ..data
          workingDir: /usr/workspace
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: workspace
              readOnly: true
              mountPath: /usr/workspace
            - name: registry-config
              mountPath: /usr/.docker
      volumes:
        - name: workspace
          configMap:
            name: source
            items:
              - key: requirements.txt
                path: requirements.txt
              - key: main.py
                path: main.py
              - key: Dockerfile
                path: Dockerfile
        - name: registry-config
          secret:
            secretName: docker-registry-img
---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-img
stringData:
  config.json: '{"auths": {"https://index.docker.io/v1/": {"auth": "xxxxxxxxxxxxxxxxxxxx"}}}'

• Makisu which can be run without custom privileges (like buildkit or img) but need to be run as root to build images. Documentation of the whole project is really good so applying it to our architecture should be relatively easy ;)

Few more data:
Image size: 20Mb
Built time: 2m23s

Input yamls:

apiVersion: batch/v1
kind: Job
metadata:
  name: makisu
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: makisu
        image: gcr.io/uber-container-tools/makisu:latest
        imagePullPolicy: IfNotPresent
        args:
        - build
        - --push=index.docker.io
        - --modifyfs=true
        - -t=wookieeteam/k3s-makisu-test:1
        - "--registry-config=/registry-config/makisu-secret.yaml"
        - /makisu-context
        volumeMounts:
        - name: context
          mountPath: /makisu-context
        - name: registry-secret
          mountPath: /registry-config
        # makisu need to be run as root -> https://github.com/uber/makisu/issues/337
        # securityContext:
        #   runAsUser: 1000
        #   runAsGroup: 1000
      volumes:
      - name: context
        configMap:
          name: source
          items:
            - key: requirements.txt
              path: requirements.txt
            - key: main.py
              path: main.py
            - key: Dockerfile
              path: Dockerfile
      - name: registry-secret
        secret:
          secretName: docker-registry-makisu
---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-makisu
stringData:
  makisu-secret.yaml: "index.docker.io:\n  .*:\n    security:\n      tls:\n        client:\n          disabled: false\n      basic:\n        username: \"xxxxxxxx\"\n        password: \"xxxxxxxxx\""

Common configmap with sources of code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: source
data:
  requirements.txt: |
    numpy
    Jinja2
    matplotlib==2.2.2
    mpld3
    flask
    gunicorn>=19,<20
  main.py: "import matplotlib\r\nmatplotlib.use(\"Agg\") \r\nimport mpld3\r\nimport
    matplotlib.pyplot as plt\r\nfrom matplotlib import colors\r\nfrom flask import
    Flask\r\nfrom flask import request\r\nimport numpy\r\n \r\ndef calculate(degree):\r\n
    \   size=3**degree\r\n    matrix=numpy.ones([size, size])\r\n    for elem in
    range(degree+1):\r\n        step=3**(degree-elem)\r\n        for x in range(size):\r\n
    \           if x%3==1:\r\n                for y in range(size):\r\n                    if
    y%3==1:\r\n                        matrix[y*step:(y+1)*step, x*step:(x+1)*step]=0\r\n
    \   return matrix\r\n\r\ndef render_carpet(degree):\r\n    matrix=calculate(degree)\r\n\r\n
    \   fig=plt.figure()\r\n\r\n    ax = plt.Axes(fig, [0., 0., 1., 1.])\r\n    fig.add_axes(ax)\r\n\r\n
    \   img=ax.imshow(matrix)\r\n    return mpld3.fig_to_html(fig)\r\n\r\napp =
    Flask(__name__)\r\n\r\[email protected]('/')\r\ndef hello_world():\r\n    degree =
    request.args.get('degree', default = 6, type = int)\r\n    print(\"Degree:\",
    degree)\r\n    return render_carpet(degree)\r\n\r\nif __name__ == '__main__':\r\n
    \   app.run()"
  Dockerfile: |-
    FROM python

    WORKDIR /usr/src/app

    COPY requirements.txt .
    RUN pip install -r requirements.txt
    COPY main.py .

    USER 1000

    CMD [ "gunicorn", "-w 4", "-b", "0.0.0.0:8000", "main:app" ]

I found more tools but every one of them does not provide any tutorial or description of how to run it in a container (ofc if is it possible):
orca-build
umoci
buildah
flt
rules_docker

[pPrecel]

Summary:
I ran all of those tools on the gardener cluster with allowPrivilegedContainers: false. The conclusion is that no one of those tools can be used in this form

[kwiatekus]

thank you @pPrecel for the investigation