IL-0104 Cisco ThousandEyes · 2025-09-04kubernetesnetworking
The talk walks through how a pod resolves a name and reaches a Service: getaddrinfo, then the KUBE-SERVICES, KUBE-SVC and KUBE-SEP iptables chains that kube-proxy writes, and conntrack, which records connection state so that packets of a known connection skip full iptables evaluation. The cluster ran Istio in sidecar mode, with Istio ServiceEntries, and the interval at which Istio refreshes its DNS cache is hard-coded.
The final slide names three causes, one each under Istio, kube-proxy and conntrack. Istio has an extremely aggressive DNS cache refresh and reuses the same client port for all DNS requests. kube-proxy had no mechanism to clean up stale conntrack entries for terminated pods; this is Kubernetes issue #126130, fixed in v1.32. conntrack's default timeout for a bidirectional UDP stream is 2 minutes.
On 4 September 2025 a CoreDNS pod in a Cisco ThousandEyes Kubernetes cluster was OOMKilled.
Symptom ? DNS resolution failuresStale endpoint routingOOMKilled
Root cause ? Software defect
–
–
–
The closing slide sets three behaviours side by side: Istio's extremely aggressive DNS cache refresh with one client port reused for all DNS requests, kube-proxy's lack of a cleanup mechanism for stale conntrack entries of terminated pods (fixed in Kubernetes v1.32), and conntrack's default bidirectional UDP stream timeout of 2 minutes. The speaker describes Istio as making some very complex things simple but some very simple things complex, and presents Cilium, which looks up Services in maps instead of walking the KUBE-SERVICES chain and can replace kube-proxy entirely with its eBPF data path.
–