IL-0056 MindTickle · 2020-03-16 (published)kubernetesawsnetworking
The team planned to move its production cluster from kops to EKS on the first weekend after New Year's, the lowest-traffic period for the platform. One of the reasons for EKS was networking, because Weave had caused a lot of trouble on the kops cluster. A new production cluster had been set up and the production apps migrated onto it. After the switch the platform became slow and unstable and the team reverted. A support case with AWS went to a networking expert.
About a month of packet captures followed. Traffic inside the cluster was fine, but calls from pods to services on EC2 behind NLBs failed intermittently, and the reachability from pods and nodes did not make sense. At the end of a call, AWS suggested setting AWS_VPC_K8S_CNI_EXTERNALSNAT=true. With that setting the calls worked as expected.
The AWS VPC CNI gives each pod its own VPC IP address using secondary addresses on the instance. Traffic within a VPC goes directly between private addresses without source network address translation (SNAT). For a destination outside the VPC, the plugin rewrites the pod's address to the primary private IP of the node's primary network interface. The NLBs were in a different VPC, so the SNAT happened on the node. The author's explanation is that the plugin translated the pod's address towards an internet gateway, and with no internet gateway in that path the traffic broke. With external SNAT enabled, the plugin does no translation and the NAT gateway translates traffic from the pod to the internet. After the change the cluster had no further networking problems, and all traffic was moved onto EKS.
On the Saturday the production cluster was moved from kops to EKS, the platform became very slow and too unstable to keep running, and the team reverted.
Packet captures requested by an AWS Support networking expert showed the pattern. Network calls inside the cluster worked. Calls from pods to microservices on EC2 machines behind an NLB (Network Load Balancer) failed intermittently. Captures taken on the sending pods, on their Kubernetes nodes and on the receiving EC2 machines showed many packet retransmissions between the node and the EC2 server.
The AWS expert read the captures as a misconfiguration on the EC2 servers and asked for socket and network statistics along with the tcpdumps. Generating traffic and collecting captures went on for a month. Then connection-check scripts showed a crossed pattern: a status endpoint was reachable from Pod-A on Node-A but not from Node-A itself, while it was reachable from Node-B but not from Pod-B on Node-B.
On a call with AWS's Kubernetes experts, changes they suggested to the node's networking module broke networking on that node completely, so no tools could be installed to debug further. In the meantime the team used an ELB, which was not showing many problems.
Root cause ? DNS or network fault
The team lead saw the platform behaving very slowly after the switch to EKS and phoned the author.
In a networking problem every component in the stack is a suspect; ruling out EKS early cost a lot of time. A DaemonSet that pinged every component from every other one exposed the anomaly that led to the fix. The team had only skimmed the AWS CNI documentation before the migration; knowing how the plugin works internally would have led to better judgements and hypotheses while debugging.
same root cause · shared AWS VPC CNI