IL-0185 Hans H. Kim · 2023-05-07 (published)kubernetes
The cluster was built with Kubespray and had three master nodes. A hardware failure took down the first control plane node. Because Kubespray treats the first node in the kube_control_plane group specially, recovering that node is very tricky.
The recovery went in steps. The control plane order in the Kubespray inventory was changed from node-1, node-2, node-3 to node-2, node-3, node-1 so that another node took the first position, and the failed machine was removed from the cluster. The cluster-info ConfigMap in kube-public was edited with kubectl edit cm -n kube-public cluster-info to replace the old control plane IP with a live node's IP and to change certificate-authority-data. A new control plane node was then prepared by hand: swap turned off, systemd-resolved started, Docker, conntrack, kubeadm and kubelet installed, the certificates in /etc/kubernetes/ssl and /etc/ssl/etcd/ssl copied over, and the CNI plugins v0.9.1 installed. It joined with kubeadm join ... --control-plane --v=5 --skip-phases control-plane-join/etcd, and the etcd member was added separately.
The author warns that losing one or more of the three control plane machines breaks the etcd quorum, so etcd has to be recovered before another node fails.
The kernel on a master node of the Kubernetes cluster logged disk I/O errors: "[HAWKEYE] kernel:: [22329228.540857] blk_update_request: I/O error, dev sda, sector 795030104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0". The sda disk switched to read-only, and the machine needed a new disk and an operating system reinstall.
The control plane ran on three master nodes for high availability, but the failed machine was the first control plane node, which Kubespray gives special privileges, so recovering it was much harder than recovering one of the other two.
Symptom ? Node NotReady
Root cause ? Hardware failure
Disk I/O errors from blk_update_request on dev sda appeared in the node's kernel log.
–
In a Kubespray cluster the first node of the kube_control_plane inventory group is special, so its failure needs its own procedure: move another node into the first position, fix the cluster-info ConfigMap, join a new control plane node and add the etcd member back. etcd should be brought back to full membership before another control plane node fails.
–