IL-0019 UK Ministry of Justice · 2023-09-18kubernetesaws
The new module version changed the launch template so that new nodes got 20 GB root volumes instead of 100 GB, and image pulls filled them. The team moved workloads back to the old node group, reviewed the launch-template diff the next day, applied a corrected template and drained again. The second drain exposed another limit: pods failed to get addresses until the old nodes released their IP prefixes. Repair took about four hours and the incident closed after a day and a half. Follow-ups cover checking launch-template changes in plans, testing image pulls at realistic sizes, scaling CoreDNS with node count and alerting on prefix starvation.
A root-volume alert fired and a user reported image pulls failing for lack of space. While nodes were being drained for an upgrade, many pods went Evicted or into an unknown container state.
Symptom ? Node disk fullImage pull failuresPod evictionIP address exhaustion
Root cause ? Configuration change
Trigger Upgrading the Terraform EKS module and draining workloads onto the new nodes it created.
A root-volume utilisation alert and a user report of disk-space errors on image pull.
A module version bump can silently change the launch template, so the plan diff needs checking before an upgrade. Draining many nodes can expose address starvation and DNS capacity gaps that need their own monitoring.
IL-0017 VPC CNI stopped allocating pod IPs on the live clustersame root cause · shared Node pool, AWS VPC CNI
IL-0017 — linked by the editor