IL-0030 Zalando · 2019-02-11 (published)kubernetesaws
Henning Jacobs of Zalando presented this as the first of five failure stories at a meetup in Hamburg on 2019-02-11. At the time Zalando counted 100 clusters and 373 accounts. This case covers only that first story; the CoreDNS outage from the same talk is a separate case.
On a cluster running Kubernetes v1.7.4 (CoreOS build), most nodes turned NotReady together, leaving fewer than 20% of nodes available. All of them recovered on their own after 15 minutes. The clues the team listed were that failure and recovery happened at the same time on every node, that the API server sat behind an AWS ELB, and that other users reported the same symptom, some of them about monthly.
The team matched it to the upstream issue kubernetes/kubernetes#48638, which was fixed in Kubernetes 1.8 and backported to 1.7.8.
Fewer than 20% of the nodes in one Zalando cluster were available. Most nodes reported NotReady at the same moment, and only a handful stayed Ready.
The kubectl get nodes output shown in the talk lists nodes such as ip-172-31-10-91...internal as NotReady, age 4d, version v1.7.4+coreos.0. In that excerpt five of eleven nodes are NotReady, four are Ready and two are Ready,SchedulingDisabled.
The nodes did not fail one by one. They all became unhealthy at the same time and all recovered at the same time, about 15 minutes later, without anyone intervening. Other Kubernetes users reported the same pattern, and some of them saw it roughly once a month.
Symptom ? Node NotReady
Root cause ? Software defect
–
–
–
same root cause · shared Cloud load balancer
same root cause · shared Kubernetes API server
same root cause · shared Cloud load balancer
same root cause · shared Kubernetes API server
same root cause · shared Cloud load balancer
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server