IL-0110 DFDS · 2024-03-13kubernetesaws
DFDS runs dev and production workloads on the same Kubernetes cluster on AWS. On 2024-03-13 a Grafana Agent pod used a large amount of memory on one node, and Hangfire pods of a dev container were scheduled to the same node. Those pods failed on system OOM, restarted and failed again in a spiral of CrashLoopBackOff until the whole node stopped responding and went NotReady. Unlike most of the monitoring stack, the Grafana Agent did not run on a dedicated node group.
In the morning the team marked the EC2 instance unhealthy so a new node started, cordoned and force-drained the NotReady node and terminated the instance, and declared the incident resolved at 12:10 while investigations continued. They then found the Grafana Agent still down because of a newly applied LimitRange, removed the LimitRange and restarted the agent. They found a correlation between the tenfold rise in EC2 network traffic, the RDS problems and the dev container, scaled the container to 0 and then back to its original 3 replicas, and the replacement node went NotReady at 15:03.
The second time, the team scaled the dev container to 0 again, a new node was added, and the team that owns the container was emailed and confirmed at 15:52 that it was working on a fix. The incident was closed at 09:20 UTC on 2024-03-14. The post-mortem gives 4 minutes to detection and 399 minutes to recovery.
At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.
CloudWatch showed a tenfold increase in EC2 network traffic leading up to the incident, and the Grafana Agent pod on the unhealthy node had high memory use. After the node was replaced, the Grafana Agent stayed down until 13:02 because of a LimitRange newly applied to its namespace.
At 13:49 the team concluded that a dev container using Hangfire to trigger jobs was failing with OOM and going into CrashLoopBackOff with new OOMs. At 13:55 they found problems on the RDS instance used by that container and a correlation between the increased EC2 network traffic, the RDS instance and the container. Scaling the container to 0 replicas brought network traffic back to normal within a few minutes, but after it was scaled back up to its original 3 replicas, the replacement node entered NotReady at 15:03.
Symptom ? Node NotReadyContainer crash loopOOMKilledMonitoring outage
Root cause ? Capacity exhaustion
A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.
Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.
same root cause · shared Container memory limit, Application database
same root cause · shared Container memory limit, Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Telemetry agent
same root cause · shared Container memory limit