IL-0015 loveholidays · 2020-02-03 (published)kubernetesgcpnetworking
In a VPC-native GKE cluster, each node is handed a /24 slice of the pod secondary range, about twice the default maximum of 110 pods, no matter how many pods it actually runs. A /16 range therefore supports 256 nodes rather than the tens of thousands of pods it seems to offer. The cluster hit that ceiling and could not grow. The team lowered max pods on node pools that run few pods, which cut address use by about a third, and moved some pools to larger instances. Expanding the secondary range was considered but not done because its behaviour was undocumented.
A deployment that usually finishes in minutes had hung for an hour: half of the new pods were serving, the other half were Pending with no node available out of 256, and the autoscaler was not adding nodes.
Root cause ? Capacity exhaustion
Trigger Ordinary growth to 256 nodes; the next scale-up failed because the pod address range had nothing left.
A colleague mentioned a slow deployment; pod events and the autoscaler status ConfigMap confirmed it.
GKE pre-allocates roughly double the max-pods count of addresses per node, so a /16 is gone at 256 nodes regardless of pod density. Max pods per pool and node size are the practical levers; bigger nodes pack better but fail bigger.
IL-0006 IP exhaustion with the AWS VPC CNI, then a warm-pool setting cascadesame root cause · shared Cluster autoscaler, VPC subnet or pod address range
IL-0007 Running out of IPv4 addresses in EKS before it became an outagesame root cause · shared VPC subnet or pod address range
IL-0016 Prometheus on the live cluster killed by memory exhaustionsame root cause · shared Node pool
IL-0013 Connection resets from a full conntrack table on GKE nodessame root cause · shared Node pool
IL-0017 — linked by the editor