IL-0188 Yonghyun Kim · 2025-11-09 (published)kubernetesaws
Kubernetes treats a pod as Ready once its containers run and its readinessProbe succeeds; it then adds the pod to the EndpointSlice and, during a rolling update, starts deleting an old pod. The ALB works on its own clock: the AWS Load Balancer Controller registers the new pod IP as a target in the initial state, and the ALB only marks it healthy after two consecutive successful health checks, at least 30 to 40 seconds later. In the meantime the old pod was already terminating and draining, the new target was not yet healthy, and the ALB had nowhere to send requests, so users got 502s for about 28 to 32 seconds.
The author solved it with pod readiness gates. Labelling the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled makes the AWS Load Balancer Controller's mutating webhook add a readiness gate to new pods; the pod stays not Ready (READINESS GATES 0/1, condition message "Target is in 'initial' state") until the controller sees the ALB target as healthy, and only then does Kubernetes delete the old pod. After the change, 10 out of 10 deployments ran without errors, and the Datadog monitor "ALB 5XX Error Rate > 5%" did not trigger for 30 days. Finding this took three days of debugging, and the incident was about two years before the post.
For the shutdown side the author also recommends a preStop sleep of 15 seconds with terminationGracePeriodSeconds of 30, lowering the target group deregistration_delay from its 300-second default to 30 seconds, and a 15-second ALB health check interval with a healthy threshold of 2. Readiness gates work only with target-type ip, make a pod take about 45 seconds instead of 10 to become Ready, and the author suggests setting the webhook failurePolicy to Ignore so a controller outage does not block deployments.
Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.
Users received "HTTP/1.1 502 Bad Gateway" pages. In Datadog the author saw lines such as "[02:34:11] ALB HealthCheck: 10.0.1.50:8080 → initial" followed by "Request → 502 Bad Gateway (no healthy targets)" at 02:34:11, 02:34:15 and 02:34:23, until "[02:34:41] ALB HealthCheck: 10.0.1.50:8080 → healthy" and a 200 OK at 02:34:42. The service was down for about 30 seconds per deploy.
Over 10 deployments, all 10 produced errors, lasting 28 seconds on average, with an average of 156 502 errors and about 50 to 100 affected users per deployment, and about three night-time pages a month.
The first attempt was to raise terminationGracePeriodSeconds to 300. Errors went down, but a rolling update of three replicas then took about 15 minutes, the application was simply killed with SIGKILL after waiting instead of shutting down cleanly, and the gap when a new pod starts was not addressed at all.
Symptom ? High error rateService outage
Root cause ? Faulty deployment
A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.
–
A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors.
Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.
same root cause · shared Ingress controller
same root cause · shared Ingress controller