IL-0082 RebelMouse · 2025-05-19kubernetesawsnetworking
CoreDNS is the DNS server that services inside a Kubernetes cluster use to find each other. During a planned reroll of the EKS nodes for security patches, the EC2 instance running one CoreDNS pod was terminated. Its share of DNS queries fell on the remaining CoreDNS pods, which could not handle the extra load, so DNS resolution slowed down and timed out for many services.
The CoreDNS deployment had been given CPU and memory of at least three times its average load. The root cause analysis found that this headroom still did not cover the sudden spike on the remaining pods when one pod disappeared. The team rescheduled the lost CoreDNS pod onto a new EC2 instance and raised the replica count of the CoreDNS deployment so the load spread evenly across more pods. Core functionality was back at 13:50 UTC, eight minutes after detection, and the incident was resolved at 14:13 UTC.
At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.
DNS lookups through CoreDNS became slow and timed out more often. Multiple services and applications were disrupted, and because this happened during peak usage hours, users widely reported the platform as unavailable.
Symptom ? DNS resolution failuresHigh latencyHigh error rateService outage
Root cause ? Capacity exhaustion
Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.
–
same root cause · shared Node pool, CoreDNS
same root cause · shared Node pool, CoreDNS
same root cause · shared Node pool, CoreDNS
same root cause · shared Node pool
same root cause · shared CoreDNS
same root cause · shared Node pool
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared CoreDNS