IL-0199 GitHub · 2020-07-13kubernetesnetworking
On 13 July 2020 GitHub.com was degraded for four hours and 25 minutes. A container that is not required for production traffic exceeded its memory limit and was killed, and because Kubernetes treats a pod as available only when every container is healthy, production pods were marked unavailable. The loss cascaded through the clusters and the reduced capacity took services down.
Normal recovery failed because the container used ImagePullPolicy Always, so Kubernetes fetched the image on every start, and a routine DNS maintenance operation finished earlier had left the clusters unable to reach the registry. A redeploy triggered to mitigate made it worse by spreading the failure across production clusters. Services recovered only after the process was restarted with the cached DNS records, which let the clusters fetch images, redeploy and come back.
At 08:18 UTC on 13 July 2020, production Kubernetes pods began to be marked unavailable. The loss of pods spread through the clusters and reduced capacity until GitHub.com services went down.
Inside the affected pods, one container was going over its memory limit and being terminated. That container was not needed to serve production traffic, but a pod is only available when all of its containers are healthy.
This kind of failure usually recovers within about a minute. This time the container had an ImagePullPolicy of Always, so each restart had to pull the image again, and the clusters could not reach the registry, so the pods failed to start. A redeploy started to mitigate the problem spread the failure to more production clusters.
Root cause ? DNS or network fault
–
A container that production traffic does not need can still take pods out of service, because a pod is available only when all of its containers are healthy.
ImagePullPolicy Always turns every container restart into a registry dependency, so an unrelated DNS change that breaks registry access turns a one-minute self-healing event into a long outage, and a mass redeploy spreads it.
–