IL-0093 UK Ministry of Justice · 2020-08-14kubernetes
The cluster ran six replicas of the ingress controller. The leader pod among them had exhausted its shared memory, and the other replicas could not connect to it to get the latest nginx configuration. Two replicas were crash-looping, and following the runbook by restarting more replicas only added more crash-looping pods, four of six in the end.
Restarting the leader ingress controller pod let the other replicas connect and fetch the latest nginx.config file, and all of them returned to ready and running. The report gives the time to repair as 37 minutes and the time to resolve as 55 minutes.
At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.
The normal runbook said to recycle all the pods. After pods 4 and 5 were restarted they also began to crash-loop, so four of the six replicas were down. Restarting the remaining pods risked leaving all six in a crash loop and every ingress in the cluster down. The incident was declared at 11:01.
The pod that had not yet been looked at or restarted was the leader ingress controller pod, and it had exhausted its shared memory.
Symptom ? Container crash loop
Root cause ? Capacity exhaustion
Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.
–
–
same root cause · shared Ingress controller
same root cause · shared Ingress controller