Incident Library?

Incident Library?

Contributing factor ?

Unreviewed defaults 기본 설정 방치 · `default-config-unchanged`

7 cases · 2019 to 2026

Root cause ?Capacity exhaustion 4 Cloud provider outage 1 Dependency failure 1 Faulty deployment 1

Component ?Cluster autoscaler 4 Node pool 3 VPC subnet or pod address range 3 AWS VPC CNI 2 Credential broker for pods 1 Descheduler 1 GKE control plane 1 Horizontal pod autoscaler 1 InfiniBand fabric 1 Ingress controller 1 Karpenter 1 Kubernetes API server 1 kubelet 1 NCCL 1 PyTorch 1

Symptom ?Pods Pending 4 IP address exhaustion 3 Service outage 2 Upgrade stall 2 Autoscaler misbehaviour 1 Scale-up failure 1 Control plane lag 1 Credentials unavailable 1 High error rate 1 Link flapping 1 NCCL timeout 1 Pod eviction 1 Training job failure 1

Cases

IL-0023 A training run failed on an NCCL timeout from one brief InfiniBand port drop — Databricks, 2026, sev4
Root cause Dependency failure
IL-0006 IP exhaustion with the AWS VPC CNI, then a warm-pool setting cascade — Neon, 2025
Root cause Capacity exhaustion
IL-0020 A descheduler evicted pods from cordoned nodes during an EKS upgrade — UK Ministry of Justice, 2025, sev1
Root cause Faulty deployment
IL-0007 Running out of IPv4 addresses in EKS before it became an outage — Adevinta, 2024, sev4
Root cause Capacity exhaustion
IL-0015 A GKE pod address range ran out at 256 nodes — loveholidays, 2020, sev3
Root cause Capacity exhaustion
IL-0002 Builder pods lost IAM credentials when kubelet fell behind the API rate limit — Zalando, 2019
Root cause Capacity exhaustion
IL-0014 A GKE upgrade that stalled, left zombie nodes and broke an ingress — loveholidays, 2019
Root cause Cloud provider outage

View on the graph