IL-0033 Airbnb · 2019-11-21 (published)kubernetesawsnetworking
Airbnb ran Kubernetes on AWS EC2 with Ubuntu images and the Canal CNI plugin (Calico with flannel), and was moving services that used to run on their own EC2 instances onto Kubernetes and containers; by September 2019 about a thousand services ran on Kubernetes. In this talk two Airbnb engineers go through latency problems that showed up during the migration.
Pods use the dnsPolicy ClusterFirst by default, so every DNS query, including queries for names outside the cluster, is sent to kube-dns through its cluster IP. That cluster IP is implemented with iptables rules, which pick a kube-dns pod at random rather than balancing load, so some kube-dns pods received far more queries than others. The team then hit a DNS packets-per-second limit, and DNS responses slowed down and timed out. The affected services all relied heavily on DNS and made many external lookups. Service latency rose from around one second to five and then 20 seconds across several services.
The speakers' answer is to choose the DNS policy per pod: a pod that does not need Kubernetes DNS resolution can use dnsPolicy Default, or None with its own DNS parameters, so that its external lookups do not pass through kube-dns.
On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.
The affected services had one thing in common: they relied heavily on DNS and sent many DNS queries for names outside the cluster. The query load on kube-dns was uneven: some kube-dns pods were handling a high rate of queries per second while others handled very little. DNS answers became slow and timed out for several services.
Symptom ? High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
Root cause ? Capacity exhaustion
Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.
–
–
By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often.
iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.
same root cause · shared Client-side DNS resolver and cache
same root cause · shared kube-dns
same root cause · shared Client-side DNS resolver and cache
same root cause · shared kube-proxy
same root cause · shared Client-side DNS resolver and cache