IL-0047 MindTickle · 2019-08-05 (published)kubernetesnetworking
MindTickle had moved most of its production workload to Kubernetes. After an earlier cluster-wide network failure that forced the team to build a new cluster and migrate every application, the team studied Kubernetes networking in depth, and in that period noticed intermittent delays in pod-to-pod calls. Because every delay was a multiple of five seconds, the team found that DNS lookups were the cause. The resolver in the C library (libc, or musl on Alpine) sends the A (IPv4) and AAAA (IPv6) queries on the same port; a race condition in conntrack, which shows up where DNAT or SNAT translation happens, sometimes loses one of them, and the resolver treats the whole lookup as timed out and retries after 5 seconds.
Adding the resolver option single-request-reopen to the pods' dnsConfig made the resolver use a different socket for the second query, and tcpdump confirmed the A and AAAA queries now left from different ports. The delays disappeared for ordinary Linux images. gRPC services still hit DEADLINE_EXCEEDED, and tcpdump showed their A and AAAA queries still sharing a port: gRPC uses its own c-ares resolver when it is available, so the libc option had no effect. Setting GRPC_DNS_RESOLVER=native in the Dockerfiles made gRPC use the system resolver, and the problem was gone there too. Alpine images were left unchanged because most of the applications use other Linux images.
Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.
gRPC services showed it as DEADLINE_EXCEEDED errors when their timeout was 3 seconds.
A capture with tcpdump port 53 made the pattern visible: the pod sent the A and AAAA queries for a name from the same source port, the AAAA answers came back, and the A query for the same name was only sent again about five seconds later (18:09:09 to 18:09:15 in the published capture) before it was answered.
Symptom ? Intermittent timeoutsHigh latencyDNS resolution failures
Root cause ? DNS or network fault
The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.
–
Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.
same root cause · shared Linux kernel, conntrack table
same root cause · shared Client-side DNS resolver and cache
same root cause · shared conntrack table
same root cause · shared conntrack table