Upgrade fails when /tmp is mounted with noexec

[hjannasch]

I have a cluster which was initialized before successfully with k0sctl Version v0.15.2 with the following config

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: itbs-training-cluster-1
spec:
  hosts:
  - ssh:
      address: 10.80.243.86
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
    role: controller
    uploadBinary: true
    k0sBinaryPath: /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
  - ssh:
      address: 10.80.243.31
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
    role: worker
    uploadBinary: true
    k0sBinaryPath: /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
    files:
      - name: bundle-file
        src: /root/zid/k0s/images/k0s-airgap-bundle-v1.28.8+k0s.0-amd64
        dstDir: /var/lib/k0s/images/
        perm: 0755
  - ssh:
      address: 10.80.243.32
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
    role: worker
    uploadBinary: true
    k0sBinaryPath: /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
    files:
      - name: bundle-file
        src: /root/zid/k0s/images/k0s-airgap-bundle-v1.28.8+k0s.0-amd64
        dstDir: /var/lib/k0s/images/
        perm: 0755
  k0s:
    version: v1.28.8+k0s.0
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: itbs-training-cluster-1
      spec:
        telemetry:
          enabled: false
        network:
          provider: kuberouter
          podCIDR: 10.100.0.0/23
          serviceCIDR: 10.100.2.0/23
        api:
          externalAddress: 10.80.243.95
          sans:
            - 10.80.243.95

After upgrading k0sctl to v0.17.5 and do a cluster upgrade I get the error spec.k0s.config fails validation. Maybe it's related to #567

Run log:

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.5 Copyright 2023, k0sctl authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
INFO ==> Running phase: Connect to hosts
INFO [ssh] 10.80.243.32:22: connected
INFO [ssh] 10.80.243.86:22: connected
INFO [ssh] 10.80.243.31:22: connected
INFO ==> Running phase: Detect host operating systems
INFO [ssh] 10.80.243.32:22: is running Red Hat Enterprise Linux 8.9 (Ootpa)
INFO [ssh] 10.80.243.86:22: is running Red Hat Enterprise Linux 8.9 (Ootpa)
INFO [ssh] 10.80.243.31:22: is running Red Hat Enterprise Linux 8.9 (Ootpa)
INFO ==> Running phase: Acquire exclusive host lock
INFO ==> Running phase: Prepare hosts
INFO ==> Running phase: Gather host facts
INFO [ssh] 10.80.243.32:22: using evdeva02.bs.ch as hostname
INFO [ssh] 10.80.243.31:22: using evdeva01.bs.ch as hostname
INFO [ssh] 10.80.243.86:22: using evdeva56.bs.ch as hostname
INFO [ssh] 10.80.243.32:22: discovered ens192 as private interface
INFO [ssh] 10.80.243.31:22: discovered ens192 as private interface
INFO [ssh] 10.80.243.86:22: discovered ens192 as private interface
INFO ==> Running phase: Validate hosts
INFO ==> Running phase: Gather k0s facts
INFO [ssh] 10.80.243.86:22: found existing configuration
INFO [ssh] 10.80.243.86:22: is running k0s controller version v1.28.4+k0s.0
WARN [ssh] 10.80.243.86:22: k0s will be upgraded
INFO [ssh] 10.80.243.32:22: is running k0s worker version v1.28.4+k0s.0
WARN [ssh] 10.80.243.32:22: k0s will be upgraded
INFO [ssh] 10.80.243.86:22: checking if worker evdeva02.bs.ch has joined
INFO [ssh] 10.80.243.31:22: is running k0s worker version v1.28.4+k0s.0
WARN [ssh] 10.80.243.31:22: k0s will be upgraded
INFO [ssh] 10.80.243.86:22: checking if worker evdeva01.bs.ch has joined
INFO ==> Running phase: Validate facts
INFO ==> Running phase: Download k0s binaries to local host
INFO ==> Running phase: Upload k0s binaries to hosts
INFO [ssh] 10.80.243.31:22: uploading k0s binary from /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
INFO [ssh] 10.80.243.86:22: uploading k0s binary from /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
INFO [ssh] 10.80.243.32:22: uploading k0s binary from /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
INFO ==> Running phase: Upload files to hosts
INFO [ssh] 10.80.243.32:22: uploading bundle-file
INFO [ssh] 10.80.243.31:22: uploading bundle-file
INFO [ssh] 10.80.243.32:22: file already exists and hasn't been changed, skipping upload
INFO [ssh] 10.80.243.31:22: file already exists and hasn't been changed, skipping upload
INFO [ssh] 10.80.243.86:22: validating configuration
INFO ==> Apply failed
FATA apply failed - log file saved to /root/.cache/k0sctl/k0sctl.log: spec.k0s.config fails validation:

[kke]

It tries to run the validation using the new binary that was uploaded to /tmp and fails to execute it. This is probably because RHEL mounts /tmp as noexec.

I can think of two workarounds:

Remount /tmp without noexec

mount -o remount,exec /tmp

And to restore:

mount -o remount,noexec /tmp

This could be put in k0sctl.yaml hooks:

spec:
  hosts:
  - ssh:
      ...
    role: controller
    uploadBinary: true
    k0sBinaryPath: /root/zid/k0s/images/k0s-v1.28.8+k0s.0-amd64
    hooks:
      apply:
        before:
          - mount -o remount,exec /tmp
        before:
          - mount -o remount,noexec /tmp

Or you could set TMPDIR env on the host to point somwhere else, like /root/tmp.

[twz123]

Can we maybe use the /run folder instead, or is it also noexec?

dir="${XDG_RUNTIME_DIR-/run}/k0sctl" && mkdir -p -- "$dir" && chmod 0700 -- "$dir" && echo "$dir"

[kke]

Or maybe the temp k0s binary should go to something like ${K0S_BINARY_PATH:-/usr/local/bin}/k0s-v1.2.3-z80+k0s.0.tmp.9999 before it is switched instead of $TMPDIR.

[hjannasch]

I will try the mount workaround and give feedback. Would be nice if you can fix the issue in a new release.

[hjannasch]

The issue is really the /tmp mount as noexec. The workaround with hooks is not working, before hooks are executed after the validation phase.

INFO [ssh] 10.80.243.86:22: validating configuration
INFO ==> Running phase: Run Before Apply Hooks
INFO ==> Running phase: Upgrade controllers

If I manually fix the mount directly on the host, k0sctrl can run the validation.