IL-0074 AirMap · 2018-07-19kubernetesnetworking
AirMap's production Kubernetes cluster, on Microsoft Azure, lost a growing number of worker nodes to "Not Ready" while pods on them crash-looped on network and DNS failures, taking every platform service down. Restarting kubelets only helped briefly. At about 18:00 UTC the team began restarting all worker nodes, staggered. Once a quorum of the nodes behind the service agent pools had been restarted, healthy API replicas were scheduled on them and the situation cleared. After about 1.5 hours every worker node had been restarted, the 5xx responses stopped, and the incident ended at 20:00 UTC.
During and after the incident the team matched the symptoms to the upstream Kubernetes issue 45419, whose root cause had been identified there along with candidate fixes. Until those fixes landed upstream and reached Azure, AirMap planned to restart nodes on a rolling schedule to keep nodes and kubelets healthy.
At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.
Most API replicas were in a restart loop and could not serve traffic, and some worker nodes were reported as "Not Ready". The per-pod and per-container logs showed network and DNS failures. The sidecar containers that connect to AirMap's internal traffic management infrastructure failed on these, and took the workload containers down with them, which caused the restart loop.
Restarting the kubelets on the stuck nodes did not hold: the nodes soon went back to "Not Ready", and more nodes followed.
Symptom ? High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage
Root cause ? Software defect
Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.
When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet