IL-0178 Lotte E-commerce · 2022-10-21 (published)kubernetesawsnetworking
Client requests to LOTTE ON reach an Application Load Balancer created from a Kubernetes Ingress, which spreads traffic over the worker nodes in its target group and on to pods through Service iptables rules that kube-proxy maintains. The pods scale with HPA and the nodes with Cluster Autoscaler, and every service is released with a blue/green deployment.
On scale-in, deleting a pod starts two things at once: the kubelet stops the containers, and the endpoints controller removes the pod from the Service endpoints, after which kube-proxy updates iptables on each node through the API server. When the container stopped before its IP left the endpoints, incoming traffic still went to it and failed with 502. Lotte E-commerce's fix was to delay container shutdown long enough for the pod to be taken out of the endpoints first. The post gives two ways to do it: wait in the application's SIGTERM handler, or, more simply, add a preStop hook that sleeps. In both cases terminationGracePeriodSeconds (30 seconds by default) caps the wait, after which the kubelet sends SIGKILL.
The blue/green errors came from connection tracking. An ALB created by an Ingress uses instance mode by default: the Service is exposed as a NodePort on every worker node and the ALB sends traffic to the nodes. Once a pod starts terminating it accepts no new HTTP connections, but the TCP connection between node and pod stays in ip_conntrack for a while, and packets that match a tracked connection are forwarded along the stored path without checking iptables. So after the Blue pods were deleted, packets on existing connections still went to their IPs and clients timed out.
The fix was to use IP mode for the ALB on some workloads. In IP mode the AWS Load Balancer Controller registers pod IPs directly as ALB targets and updates the target group as soon as pods are added or removed, so traffic goes straight to pods without iptables or conntrack, with fewer network hops. A target group holds at most 1000 targets; Lotte E-commerce's services stayed under that, and it planned to move all workloads to IP mode gradually.
LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.
A second problem appeared after blue/green deployments. The team created the Green Deployment, switched the Service label selector to it, confirmed it worked and started deleting the Blue pods. While the Blue pods' lifecycle hooks were running, the service still worked, but once the pods were fully removed, clients started getting timeout errors because part of the traffic was still being sent to the deleted pods.
On the nodes, the iptables rules no longer had any entry for the deleted pod's IP, but ip_conntrack still held the connection in the established state, and client requests kept flowing to the removed pod.
Symptom ? High error rateIntermittent timeoutsStale endpoint routing
Root cause ? DNS or network fault
–
Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application.
With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.
same root cause · shared conntrack table, kube-proxy
same root cause · shared conntrack table
same root cause · shared conntrack table