IL-0048 unknown · 2025-08-12 (published)kubernetesawsnetworking
Jack Lindamood describes an investigation in a very large AWS EKS cluster. Intermittent DNS failures in CI were traced to CoreDNS forwarding too many uncached queries to the VPC's Route 53 Resolver. Traffic to that resolver goes to a link-local address, and EC2 drops packets above 1024 per second on such traffic per network interface; the linklocal_allowance_exceeded counter in ethtool confirmed the drops.
Packet captures showed that most of the uncached traffic was for EC2-style hostnames, each queried five times. Pods get a resolv.conf with ndots:5 and four search domains (app-namespace.svc.cluster.local, svc.cluster.local, cluster.local, us-east-2.compute.internal), so any name with fewer than five dots is first tried with every search domain appended. The query pattern revealed the namespace and that the source pod used host networking, which led to one administrative pod that issued A and AAAA queries for its node's hostname every few seconds.
By cutting parts of the pod's main loop, the author narrowed the traffic down to a few subprocess calls that ran commands with sudo. Running the same commands without sudo produced no DNS traffic. An ltrace of sudo showed libsudo_util.so calling gethostname and sudoers.so calling getaddrinfo on the node's hostname, so sudo itself made the lookups because of its fqdn default. Adding 'Defaults !fqdn' to the container image stopped the lookups, and uncached DNS queries to the VPC resolver almost disappeared.
CI jobs failed now and then because a hostname could not be resolved. The failures came from tensorstore, appeared on several different instances and did not repeat consistently. The error read: "./tensorstore/util/result.h:420: Status not ok: status(): UNAVAILABLE: CURL error Could not resolve hostname: Could not resolve host: company-bucket.s3.amazonaws.com".
The CoreDNS logs (kubectl logs -n kube-system deployment/coredns --tail=1000 | grep -i error) showed upstream queries timing out on the way to the VPC's Route 53 Resolver at the VPC+2 address, for example "[ERROR] plugin/errors: 2 company-bucket.s3.amazonaws.com. A: read udp 10.1.2.3:44211->10.0.0.2:53: i/o timeout". Most failing lookups were for internal hostnames such as ip-10-8-1-3.us-east-2.compute.internal, and many were AAAA queries, so every name was being looked up twice.
AWS support pointed to a hard limit of 1024 packets per second for traffic to link-local services, which includes the VPC DNS resolver. Running ethtool -S eth0 | grep linklocal_allowance_exceeded on an affected node returned a non-zero count. Because DNS runs over UDP, CoreDNS never learned that packets were dropped; it only saw timeouts.
A tcpdump of port 53 on a CoreDNS node showed, after the expected traffic, a long tail of host-name-like queries, each repeated with every search domain appended: ip-10-0-1-10.us-east-2.compute.internal.app-namespace.svc.cluster.local, ...svc.cluster.local, ...cluster.local, ...us-east-2.compute.internal.us-east-2.compute.internal, and finally the name itself. These were cache misses that CoreDNS forwarded to the VPC resolver.
Root cause ? Capacity exhaustion
Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o timeout errors in the CoreDNS logs.
–
Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per interface, and drops show up only as UDP timeouts; the linklocal_allowance_exceeded counter in ethtool shows them. Names that are not fully qualified are multiplied by the search domains under ndots:5, and AAAA lookups double them again. sudo looks up the host's own name on every call unless 'Defaults !fqdn' is set.
same root cause · shared Client-side DNS resolver and cache, CoreDNS
same root cause · shared Client-side DNS resolver and cache, CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared Client-side DNS resolver and cache