IL-0203 Honeycomb · 2024-04-09kubernetesaws
On April 9, 2024, event ingestion and the UI of Honeycomb's EU1 instance were unavailable from 22:00:10 to 22:11:30 UTC. Earlier that day Honeycomb had begun rolling out a routine system update to its Kubernetes cluster. The update had been tested extensively in development and staging, but in production it interacted badly with other components of the cluster and made the AWS Load Balancer Controller and other pods fail sporadically.
With the controller down, new pods were no longer registered with the ALB. The next deployment replaced the old pods one at a time, the ALB kept sending traffic only to the old pods, and when the last of them was terminated the ingestion service and the UI service lost all their traffic. Honeycomb restarted the AWS Load Balancer Controller to restore service and rolled back the system update so the failure would not come back. The exact cause of the interaction was still being investigated when the write-up was posted.
Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.
The outage came at the end of an ordinary deployment. Honeycomb deploys by terminating pods and starting new ones one at a time. The new pods were working, but no traffic was forwarded to them; service broke off when the last old pod was terminated.
The reason was that the AWS Load Balancer Controller, which registers each new pod with the Application Load Balancer (ALB) so the ALB starts sending it traffic, had failed earlier. The routine system update rolled out to the cluster that day made the controller and other pods fail sporadically.
Symptom ? Service outageStale endpoint routing
Root cause ? Breaking change in an upgrade
The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.
A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Cloud load balancer
same root cause · shared Ingress controller
same root cause · shared Ingress controller