IL-0170 Backpackr · 2025-05-08 (published)kubernetesawsnetworking
Backpackr runs the handmade marketplace idus and the crowdfunding platform Tumblbug on Amazon EKS. Every external and internal request went through the Istio ingressGateway on a dedicated Istio node, and as services grew and domains were split, more and more traffic depended on that node. nf_conntrack, the Linux kernel module that records and tracks network connections, is active in container environments for NAT and port forwarding, and once its table (65536 entries by default) is full, newly arriving packets are dropped.
During a Lunar New Year event in January 2024, traffic rose to more than three times normal. The nf_conntrack table on the Istio node overflowed, packets were dropped, the ingress gateway stopped answering, and the web and app services went down completely. The Istio node had also hit its network performance limit. Backpackr names the root cause as the network performance of the Istio node, more precisely the nf_conntrack table. The event was stopped.
Afterwards Backpackr raised net.netfilter.nf_conntrack_max on every node to (total instance memory in bytes)/16384/2, and to three times that on Istio nodes, which gives about 500,000 on a 16 GB instance. A nodeOptimizer DaemonSet applies the value automatically, using the instance metadata security groups to tell Istio nodes apart. Backpackr also set up mesh networking so that sidecars talk to each other directly over HTTP instead of through the ingressGateway (HTTPS calls still need TLS offloading at the ingressGateway or NLB), moved the Istio nodes from c6.xlarge (12.5 Gbps) to an n-type instance with up to 25 Gbps, and strengthened monitoring with ENA network bandwidth and packet drops, detailed Istio metrics for the ingressGateway and sidecars, and CPU, memory and saturation.
During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.
Traffic was more than three times the usual level. All external and internal traffic passed through the Istio ingress gateway on a dedicated Istio node, and on that node the nf_conntrack table reached 100 percent of its size. New packets were dropped and the ingress gateway failed to respond. The node had also reached the limit of its network performance.
Symptom ? Service outage
Root cause ? Capacity exhaustion
A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.
When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.
same root cause · shared Ingress controller
same root cause · shared conntrack table
same root cause · shared Ingress controller