IL-0027 Backpackr · 2025-05-07 (published)kubernetesawsnetworking
Backpackr runs the idus handmade marketplace and the Tumblbug crowdfunding platform on Amazon EKS. CoreDNS on the production cluster still had the setup it got when the cluster was created: two pods, kept on the on-demand node group with a nodeSelector, and no other tuning. Every pod also used the Kubernetes default ndots:5 in /etc/resolv.conf, so any name with fewer than five dots was first tried with each search domain appended. A lookup of idus.com went out as up to five queries (idus.com.idus-backends.svc.cluster.local, idus.com.svc.cluster.local, idus.com.cluster.local, idus.com.ap-northeast-2.compute.internal, and finally idus.com), and only the last one could succeed. Under production traffic this multiplied the load on CoreDNS, which showed up as slow or failed lookups and intermittent connection failures to RDS and Redis. The problem was much more visible in production than in the dev and stage environments.
The outage came when both CoreDNS pods ended up on one node. The pods had no memory limit and ran in the BestEffort QoS class. When the node's memory usage spiked, the node ran out of memory, the kubelet stopped, and all pods on the node fell into CrashLoopBackOff. Because there was no CoreDNS pod on any other node, the cluster lost DNS and the service went down completely.
Backpackr fixed it by putting NodeLocal DNSCache in front of CoreDNS as a DaemonSet that caches answers on every node, by spreading CoreDNS with a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones (both are now EKS defaults; older EKS versions did not set them), and by lowering ndots from 5 to 2 through dnsConfig in the Deployments. Appending a trailing dot to every domain name was rejected because every reference in code and configuration would have had to change. Istio DNS proxying was tested but not adopted because it overlaps with NodeLocal DNSCache. The write-up also suggests running CoreDNS on its own node group or on Fargate and using affinity and priorityClassName.
Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.
The team first checked application code, node resources, pod status and container settings. A closer look showed that the failing calls were failed DNS resolutions. Tuning the CoreDNS cache and its bufsize setting eased the symptoms for a while but did not remove them.
The worst episode was a full outage. At one point every CoreDNS pod was running on the same on-demand node. That node's memory usage jumped until the node ran out of memory, the kubelet stopped working, and every pod on the node went into CrashLoopBackOff. With no CoreDNS pod left to answer, DNS stopped working and the whole service went down.
Symptom ? DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage
Root cause ? Capacity exhaustion
The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.
–
The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale.
Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.
same root cause · shared Node pool, CoreDNS, Container memory limit
same root cause · shared CoreDNS, Client-side DNS resolver and cache, Container memory limit
same root cause · shared Node pool, CoreDNS
same root cause · shared CoreDNS, Client-side DNS resolver and cache
same root cause · shared Node pool, CoreDNS
same root cause · shared Node pool, Container memory limit
same root cause · shared kubelet, Container memory limit
same root cause · shared Node pool, Container memory limit
same root cause · shared Node pool, Container memory limit
same root cause · shared kubelet
same root cause · shared Node pool
same root cause · shared CoreDNS
same root cause · shared Container memory limit
same root cause · shared CoreDNS
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared kubelet
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Client-side DNS resolver and cache
same root cause · shared Container memory limit
same root cause · shared kubelet