IL-0038 Oath · 2018-05-04 (published)kubernetes
Oath ran Kubernetes on premises for media sites such as its sports, finance and home pages, in 12 production clusters across six regions behind anycast routing. While rolling out a kubelet change in one cluster, kubelet hit a Kubernetes bug during its cgroup cleanup at restart and did not come back. The rolling update kept going batch by batch with no check that the updated kubelets worked, so it reached the whole cluster.
With every kubelet down, the API server saw every node as NotReady and the node controller began evicting pods. Its built-in protection limited the damage: above 50% NotReady nodes it slowed eviction, and with all nodes down it entered full disruption mode and stopped evicting altogether. About half the pods had been evicted by then; the rest kept serving because the pods and the ingress path were healthy. The team failed the data center over so users were served from the next one.
During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.
The graphs showed a growing number of nodes entering the eviction queue and pods being evicted from them. The node controller in the Kubernetes controller manager tries to move pods off a NotReady node so they can run elsewhere, but with no Ready nodes left the evicted pods could only sit in Pending.
The workloads themselves were still running: only the link between kubelet and the API server was broken, and traffic from the ingress to the pods still worked. Once half of the nodes were NotReady, the node controller slowed its eviction rate, and when it saw the whole cluster down it entered full disruption mode and stopped evicting. By then about half of the pods had been evicted, so users saw a degraded service rather than a full outage.
Symptom ? Node NotReadyPod evictionPods Pending
Root cause ? Software defect
Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.
Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet