k0sctl is not handling node removal correctly #603
Comments
|
Additionally, it seems like adding The attached zip has 3 phases: bootstrap, reset leader and remove leader. All with input |
It just goes through all controllers in the config and picks the first that has k0s running and isn't marked to be reset. If one can't be found, the first controller is used as "leader". There shouldn't be any special treatment for the leader, it's just a "randomly" picked controller that is used for running commands that need to be run on a controller. // Pick the first controller that reports to be running and persist the choice
for _, h := range controllers {
if !h.Reset && h.Metadata.K0sBinaryVersion != nil && h.Metadata.K0sRunningVersion != nil {
s.k0sLeader = h
break
}
}
// Still nil? Fall back to first "controller" host, do not persist selection.
if s.k0sLeader == nil {
return controllers.First()
}
Hmm, validated how?
The ControlNode objects are autopilot's, so it seems deleting a kubernetes node does not trigger a removal from autopilot, I don't know how autopilot manages removed nodes. I think k0sctl should maybe do This is btw automatically done when needed: environment:
ETCD_UNSUPPORTED_ARCH: armYour arch seems to be arm64, 64-bit arm is supported on etcd 3.5.0+ which is included in k0s v1.22.1+k0s.0 and newer |
|
I just removed a controller node by:
result: it removed the node from k8s, reset the host, but it did not remove the etcd member. I manually removed the member afterwards with |
It should pick a controller that is already running as the leader unless none of them are running, in which case the first is chosen: k0sctl/pkg/apis/k0sctl.k0sproject.io/v1beta1/cluster/spec.go Lines 58 to 69 in 0fb74be
k0sctl/phase/reset_controllers.go Lines 90 to 101 in 0fb74be
This could be a bug, does the
k0s just removed usage of machine id: k0sproject/k0s#4230 and that is reflected in k0sctl: #697 - so no more MachineID. For the root problem of performing apply when a fresh machine has replaced one that existed previously with the same IP, there needs to be some check for this. K0sctl should get a list (or maintain a list on its own that is distributed to all of the controllers?) of controllers known to exist and either error out and refuse to apply or try to somehow solve the situation (just do a |
Letting |
|
k0sctl already does |
|
Looks like there's another bug when a new controller is introduced, unsure if it's always or only when it replaces a previously existing one. It seems the apply fails with "empty content on file write" when k0sctl writes k0s configs on the still existing controllers. |
|
The issue is still there on k0s Install and prepare for testsLocal installation via bootloose (hence weird ssh addresses) with haproxy LB for control plane set as before; ---
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0sbootloose
spec:
hosts:
- ssh:
address: 127.0.0.1
user: root
port: 12002
keyPath: ./cluster-key
role: controller
hostname: c0
environment:
ETCD_UNSUPPORTED_ARCH: arm
reset: false
- ssh:
address: 127.0.0.1
user: root
port: 12003
keyPath: ./cluster-key
role: controller
hostname: c1
environment:
ETCD_UNSUPPORTED_ARCH: arm
reset: false
- ssh:
address: 127.0.0.1
user: root
port: 12004
keyPath: ./cluster-key
role: controller
hostname: c2
environment:
ETCD_UNSUPPORTED_ARCH: arm
reset: false
- ssh:
address: 127.0.0.1
user: root
port: 12005
keyPath: ./cluster-key
role: worker
privateAddress: 192.168.67.211
hostname: w0
reset: false
- ssh:
address: 127.0.0.1
user: root
port: 12006
keyPath: ./cluster-key
role: worker
hostname: w1
reset: false
- ssh:
address: 127.0.0.1
user: root
port: 12007
keyPath: ./cluster-key
role: worker
hostname: w2
reset: false
k0s:
version: v1.30.1+k0s.0
dynamicConfig: false
config:
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
creationTimestamp: null
name: k0sbootloose
spec:
api:
externalAddress: k0s-lb0
sans:
- 127.0.0.1
- k0s-lb0For reference, this is the localhost port mapping and internal network IP list: Installation from scratch works fine, verified by re-running Tested cluster health with: for port in 12002 12003 12004; do ssh [email protected] -p $port -i cluster-key -- "echo -n \$HOSTNAME' '; k0s etcd member-list"; doneReported expected results: Then, the first controller that's usually a leader ( Scenario 1 - controller VM reappears, attempt to use k0sctl on the same yamlImmediately after removal, VM is re-created (using After that, the first Test command shows that c0 has no etcd running, but c1 and c2 still expect c0 as cluster member: After that, Split brain is further confirmed by etcd test command: Scenario 2 - controller VM reappears, attempt to use k0sctl with reset flagImmediately after removal, VM is re-created (using After that, the config is changed to swap reset flag on re-created controller to be true. That log still mentions only removal from k0stcl config and not https://docs.k0sproject.io/stable/remove_controller, so the operator could assume everything is fine. However, c0 is NOT removed from etcd cluster that runs on c1+c2: After reverting reset flag, Cluster status in unchanged: An attempt to re-run |
|
#714 is an attempt to fix some of that. |
|
v0.18.0 solves the issue with Additionally, without |
Versions
v1.28.3+k0s.0v0.16.0(7e8c272)References
Summary
k0sctl silently does not support situation where controller entry disappears from k0sctl.yaml
spec.hosts. Additionally, k0sctl treats k0sctl.yaml as truth source and does not check every controller. In worst-case scenario, it leads to split-brain: 2 clusters exist under mandatory HA for Control Plane. It is especially visible when workflows like Terraform expect controller removal to be a single run.Details
All situations assume that the starting point is a set of 3 VMs for controllers (c01, c02, c03) and 4 VMs for workers (w01, ...) that have static IPs assigned. All VMs are fresh EC2s started from latest Ubuntu AMIs before every scenario is started. Static IPs ensure that ETCD problems are immediately visible. First controller is always targetted to ensure any issues with leadership are immediately visible.
Cluster health and membership can be verified by running
k0s etcd member-liston each VM or checking if each controller can usek0s kubectlto obtain information about the cluster.Scenario 1: procedure to leave ETCD is followed -> works
A cluster is created using
k0sctl applyand verified to be working. If you follow the procedure tok0s etcd leavetargetting leader and executek0sctl applyright away, it'll fail, but the cluster is left intact.After that, if you complete the full procedure to finish controller removal by executing
k0s stop; k0s reset; rebootand leader and then runk0sctl apply, it'll work as expected by adding c01 back to the cluster. All operations are verified to be OK.Scenario 2: controller is removed from k0sctl.yaml without any additional operations -> fails, but not catastrophically
A cluster is created using
k0sctl applyand verified to be working. If you remove leader c01 from YAML file and runk0sctl applyit seems like it was removed from the cluster (e.g. log says, "2 controllers in total").However, all 3 controllers remain in cluster and there's no outage.
If you re-add c01 entry to the hosts list in the same form as before and run
k0sctl apply, it is "added back" to the cluster (log says "3 controllers in total"). Nothing changed and the cluster is still having 3 controllers and working fine.Scenario 3: controller is externally wiped and k0sctl runs on unchanged file -> breaks cluster
A cluster is created using
k0sctl applyand verified to be working. Leader VM is destroyed by external means, user may not be aware of that. Some new fresh VM exists with the same IP address (hostname may be different).k0sctl applyis executed on YAML file that wasn't changed or changes were made, butsshsection remains the same (i.e. hostname, environment could have been changed, for example by Terraform when EC2 is rebuilt). The effect is the following:This seems to be solved in the current main branch (
39674d59b2f9546f83c74127dd64fb9dd553fad5), but only lowers severity to "fails, but not catastrophically". Re-runs of the command do not trigger recently addedetcd leave.Actual problems in one list
k0s etcd leave) in such a way that real-world matches k0sctl.yaml before running "apply"v0.16.0, only the host that used to be leader when k0sctl was last run is validatedMatadata.MachineIDssh.addressto hostname (which you can make globally unique) then ETCD cannot startThe text was updated successfully, but these errors were encountered: