IL-0011 Skyscanner · 2019-04-23 (published)kubernetesnetworking
The controller shuffled its backend map correctly, but the configuration template iterated that map with the language's range construct, which visits keys in sorted order. Every HAProxy instance therefore listed pods sorted by address and, after each hard reload, restarted round-robin from the same first pod, so one pod took the brunt. A load test on a static cluster had not reproduced the reloads that production saw. The stop-gap was pointing the service back at its virtual address; the fix kept an ordered slice and iterated over that, and an upstream fix followed in the next release.
On a Sunday a cluster user paged support about high tail latency. A few pods of the service had full request queues and refused work while others sat idle, and several ingress controller containers were being killed for memory, which pushed more load onto the rest.
Symptom ? High latencyLoad imbalanceOOMKilledIntermittent connection failures
Root cause ? Software defect
Trigger Upgrading the HAProxy ingress controller and switching backends from service addresses back to pod addresses, trusting a new default that was believed to shuffle backends.
A cluster user paged out-of-hours support; Prometheus queries confirmed the uneven distribution.
Test changes to cluster components against the exact behaviour that matters, under production-like churn. A keyword does not behave the same in every part of a language, and a small component can hurt everyone on the cluster.
–