IL-0057 Prezi · 2019-10-03 (published)kubernetesawsnetworking
When a pod connects to an external service, meaning one with a public IP, the outgoing packet goes through source network address translation (SNAT) on the worker node, set up by iptables rules. A race condition in the Netfilter module sometimes drops SYN packets while new connection entries are inserted into the conntrack table, which leads to connection timeouts of 1 or 3 seconds. A search for "kubernetes slow outgoing connections" led Prezi to a XING article describing exactly these symptoms.
The fix that article proposes, switching the port allocation to fully random, was not usable on EKS: the amazon-vpc-cni-k8s plugin supports AWS_VPC_K8S_CNI_RANDOMIZESNAT=prng, but the iptables version in the official AMI did not. Prezi instead set AWS_VPC_K8S_CNI_EXTERNALSNAT=true, which turns off SNAT on the node altogether. The write-up calls the EKS documentation for that option misleading and points to an open pull request about it.
After the setting was applied on 8/15, latency returned to the baseline from before the proxy migration. Some improvement from 8/10 came only from running more pods on more nodes, so each pod saw less traffic and hit the race less often.
Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.
The slow requests were in services that make many sequential service-to-service calls, which at Prezi also pass through the proxy. Stress tests reproduced it. AWS X-Ray traces, added for this investigation, showed a call chain smartgateway → prezipage → smartgateway → presentationservice taking 1.3 seconds, of which a full second was spent between smartgateway and presentationservice, while presentationservice itself answered in 12.8 ms.
The extra delay was always exactly 1 second or, less often, 3 seconds. A custom trace attribute carrying Nginx's $upstream_connect_time confirmed that the time went into opening the outgoing TCP connection.
Symptom ? High latencyIntermittent timeouts
Root cause ? DNS or network fault
Prometheus alerts on p99 latency fired after the reverse proxy migration.
–
Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.
same root cause · shared Linux kernel, conntrack table
same root cause · shared conntrack table
same root cause · shared AWS VPC CNI
same root cause · shared conntrack table