Check new controllers against etcd member-list to detect replaced hosts

[kke]

Addresses #603

Before installing new controllers, check if the address is already found from k0s etcd member-list. If a match with the same address is found, the apply process is aborted. K0sctl can try to perform the etcd leave for the vanished address automatically when --force is used. This check is skipped when kine or an external etcd is used.

This protects against situations when a controller host in k0sctl.yaml gets completely wiped or replaced with a fresh one between two runs. Previously k0sctl would have treated it as a regular new controller and caused the etcd to get confused / split brained.

There was also a bug when adding new controllers in general, writing the config on the previously existing clusters could have failed with "empty content on file write" error.

[kke]

This was confirmed to help with the problem in some conditions.

Could we also use the node list as another way to see if the node is previously known maybe?

[twz123]

Could we also use the node list as another way to see if the node is previously known maybe?

By node list, you mean, figuratively speaking, kubectl get nodes? That will only show k0s nodes that run the worker components. Controller-only k0s nodes won't show up.

The best way of checking etcd memebers would be to use the EtcdMemeber CRDs. You probably want to try that first. This only works for new k0s versions, but in case the CRDs aren't registered, k0sctl could gracefully degrade and use k0s etcd member-list. Another possibility would be to try the Autopilot ControlNode CRDs. Although they're not perfect, too, and there may be 🐉 🐉 🐉.