validation phase

[till]

Was wondering if the validation phase could take things like kine into account? As in test connecting to the database. Not sure if there's a utility to leverage so it wouldn't lead to writing code for all kinds of adapters?

Some context:

I copied the wrong password into the yaml configuration and apply just sat there. When I cancelled, I am stuck in a loop where k0sctl is trying to acquire an exclusive lock.

[kke]

is trying to

It is renewing the "lease" on a timer so when nothing happens on the remote, all you see is it refreshing the locks in the background.

More timeouts is for sure something k0sctl needs.

[till]

I looked into this this morning, so basically the k0scontroller log had errors about kine (expected since I botched the password) and kine constantly restarted but never finished. Never ending loop kind of.

Reset also didn't work on the cluster anymore, because of the locks. So I wrote some ansible to clean-up all the hosts.

---

This morning I re-ran apply on my new cluster after cleaning it up, it did not go through for some reason, but it exited:

INFO ==> Running phase: Initialize the k0s cluster 
INFO [ssh] 10.108.1.129:22: installing k0s controller 
INFO [ssh] 10.108.1.129:22: waiting for the k0s service to start 
INFO [ssh] 10.108.1.129:22: waiting for kubernetes api to respond 
INFO ==> Running phase: Install controllers 
INFO [ssh] 10.108.1.129:22: generating token      
INFO [ssh] 10.108.1.210:22: writing join token    
INFO [ssh] 10.108.1.210:22: installing k0s controller 
INFO [ssh] 10.108.1.210:22: starting service      
INFO [ssh] 10.108.1.210:22: waiting for the k0s service to start 
INFO * Running clean-up for phase: Acquire exclusive host lock 
INFO * Running clean-up for phase: Initialize the k0s cluster 
INFO * Running clean-up for phase: Install controllers 

(I'll look into what the problem exactly was later.)

When I try to re-run apply:

INFO ==> Running phase: Gather k0s facts 
INFO [ssh] 10.108.1.129:22: is running k0s controller version 1.27.5+k0s.0 
INFO * Running clean-up for phase: Acquire exclusive host lock 
ERRO apply failed - log file saved to /root/.cache/k0sctl/k0sctl.log 
FATA failed on 1 hosts:
 - [ssh] 10.108.1.210:22: k0s doesn't appear to be running but has been installed as a service at /etc/systemd/system/k0scontroller.service - please remove it or start the service 

Seems the clean-up forgets files?

[till]

Fixed another issue (forgot to assign a FIP to my load balancer), now it goes further.

I noticed this:

INFO ==> Running phase: Install workers  
INFO [ssh] 10.108.1.50:22: validating api connection to https://fip:6443 
INFO [ssh] 10.108.1.128:22: validating api connection to https://fip:6443 
...

I think that should also run for when other controllers join. Would make it a bit more user friendly to figure out what didn't work when the controller cannot join. :)

[kke]

Seems the clean-up forgets files?

Kind of yes. K0sctl runs k0s install, which succeeds. When giving up waiting for it to wake up, it doesn't run reset to clean up.

Maybe any errors that come after k0s install has been run should come with a note like "hosts were left in a bad state, you may want to run 'k0s reset' before trying again".