IL-0081 Cresta · 2025-03-05 (published)kubernetesnetworking
Cresta runs the NGINX Ingress controller in front of its Kubernetes services. NGINX applies big configuration changes with a full reload, which starts new worker processes and drains the old ones, and applies backend endpoint changes dynamically inside the running workers without a reload.
Cresta had enabled enable-serial-reloads, which allows only one full reload at a time so that a burst of configuration changes does not start too many worker processes. The intended behaviour is that endpoint updates still go through immediately. Because of a bug, while workers are reloading the controller returns an error early and stops all processing, so dynamic endpoint updates wait until the running reload has finished. During a rollout this meant traffic could be sent to outdated or terminated pods, and some API requests timed out.
An engineer restarted the ingress controller while trying to resolve the problem, and an issue was opened in the kubernetes/ingress-nginx project as a result of the incident.
During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.
Endpoint changes from the rollout were not applied until a running NGINX reload had finished, so requests could be sent to outdated or already terminated pods.
Symptom ? Intermittent timeoutsStale endpoint routingHigh latency
Root cause ? Software defect
–
A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller