IL-0001 Zalando · 2019-01-07kubernetesaws
CoreDNS pods ran with a 100Mi memory limit set from old usage figures. A spike of DNS queries pushed them over the limit, they were OOM-killed, and under sustained load they kept dying as soon as they came back. Nothing raised the limit automatically. The application side made it worse: the Node.js resolver did no caching, the default ndots of 5 multiplied lookups, and connections were short-lived. On-call learned about it by phone because the alerting that would have paged them lived inside the broken cluster. Raising CoreDNS requests and limits by hand to 2000Mi ended the outage.
Product pages on the storefront returned errors for more than an hour. Every DNS lookup inside the cluster failed, and because the cluster's own monitoring was also down, nobody was paged automatically.
Symptom ? High error rateDNS resolution failuresOOMKilledMonitoring outage
Root cause ? Capacity exhaustion
Trigger Calls from the aggregation layer to one downstream service timed out; clients retried, and the retry burst turned into a burst of DNS queries against CoreDNS.
An on-call engineer from the aggregation layer phoned the Kubernetes on-call engineer; automatic paging depended on in-cluster monitoring that was down.
DNS in the cluster was not resilient enough: one application could take it down for everyone. Monitoring that runs inside the cluster fails together with the cluster.
IL-0016 Prometheus on the live cluster killed by memory exhaustionsame root cause · shared Container memory limit
IL-0005 — linked by the editor