IL-0006 Neon · 2025-05-16kubernetesaws
The region's subnets were half the size used elsewhere, and growth pushed them to their limit. The VPC CNI kept many addresses warm on nodes that could not schedule more pods, so the pool looked emptier than it was. Changing the warm-pool settings helped at first, but a misread of the documented defaults then stalled pod starts after every pod deletion because released addresses sat in a cooldown that counted against the target. Setting the warm targets to zero stabilised the cluster; reverting the change a few days later reproduced the failure. The team enlarged pre-created compute pools to hide the remaining errors while the control-plane job was fixed.
New compute pods were scheduled but could not start, failing with IP allocation errors. The subnets looked exhausted even though many addresses seemed unused, and idle customer databases could not be woken.
Symptom ? Pods PendingIP address exhaustion
Root cause ? Capacity exhaustion
Trigger A control-plane job that terminates idle computes started failing, so active computes doubled over a weekend; three days later a revert of a CNI setting brought the same symptoms back.
IP assignment errors and a falling pod-start rate in metrics and logs.
Defaults and documentation need verifying under real pod churn, a revert does not necessarily restore a healthy state, and capacity assumptions from load tests break when growth outpaces them.
IL-0007 Running out of IPv4 addresses in EKS before it became an outagesame root cause · shared VPC subnet or pod address range, AWS VPC CNI
IL-0015 A GKE pod address range ran out at 256 nodessame root cause · shared VPC subnet or pod address range, Cluster autoscaler
IL-0007 — linked by the editorIL-0015 — linked by the editorIL-0017 — linked by the editor