Incident Library?

Incident Library?

Root cause ?

Capacity exhaustion 용량 한계 · `capacity`

7 cases · 2019 to 2025

Component ?Node pool 3 VPC subnet or pod address range 3 Cluster autoscaler 2 Container memory limit 2 AWS VPC CNI 2 conntrack table 1 CoreDNS 1 Credential broker for pods 1 Client-side DNS resolver and cache 1 HAProxy 1 Karpenter 1 Kubernetes API server 1 kube-proxy 1 kubelet 1 Prometheus 1

Contributing factor ?Unreviewed defaults 4 Unknown limit 3 No capacity planning 3 Monitoring blind spot 1 No alert on limit 1 No caching 1 Retry storm 1 Ineffective rollback 1 Single point of failure 1

Symptom ?High error rate 3 IP address exhaustion 3 Pods Pending 3 Monitoring outage 2 OOMKilled 2 Scale-up failure 1 Connection resets 1 Control plane lag 1 Credentials unavailable 1 DNS resolution failures 1 Intermittent connection failures 1 Node NotReady 1

Cases

IL-0006 IP exhaustion with the AWS VPC CNI, then a warm-pool setting cascade — Neon, 2025
IL-0007 Running out of IPv4 addresses in EKS before it became an outage — Adevinta, 2024, sev4
IL-0016 Prometheus on the live cluster killed by memory exhaustion — UK Ministry of Justice, 2023, sev3
IL-0013 Connection resets from a full conntrack table on GKE nodes — loveholidays, 2020
IL-0015 A GKE pod address range ran out at 256 nodes — loveholidays, 2020, sev3
IL-0002 Builder pods lost IAM credentials when kubelet fell behind the API rate limit — Zalando, 2019
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019

View on the graph