IL-0007 Adevinta · 2024-10-22 (published)kubernetesaws
The VPC CNI gives every pod an address from the VPC, so pod growth consumes IPv4 space directly. The first failure hit a small, barely used cluster and had little impact, but the same would have happened fleet-wide. The platform team rolled out custom networking with a secondary CIDR, first on small clusters and then everywhere after pausing the migration. Months later, moving to Karpenter brought the shortage alerts back because the new provisioner chose smaller instances and ignored custom networking; a reserved-ENI setting and an upstream change fixed that.
A pager alert reported DaemonSets failing on a small cluster. Pod status showed the CNI could not assign an address to new containers.
Symptom ? IP address exhaustionPods Pending
Root cause ? Capacity exhaustion
Trigger unknown; growing pod counts during a large EKS migration pushed one cluster's subnets to their limit.
A pager alert on failing DaemonSets, then pod status on the node.
The VPC CNI needs capacity planning before it is ready for a large fleet; what works in a small cluster does not hold at scale.
IL-0006 IP exhaustion with the AWS VPC CNI, then a warm-pool setting cascadesame root cause · shared AWS VPC CNI, VPC subnet or pod address range
IL-0015 A GKE pod address range ran out at 256 nodessame root cause · shared VPC subnet or pod address range
IL-0015 — linked by the editorIL-0019 — linked by the editor