Incident Library?

Incident Library?

Symptom ?

Upgrade stall 업그레이드 정체 · `upgrade-stuck`

3 cases · 2019 to 2025

Root cause ?Cloud provider outage 1 Configuration change 1 Faulty deployment 1

Component ?Cluster autoscaler 2 GKE control plane 2 Node pool 2 Admission webhook 1 Descheduler 1 Horizontal pod autoscaler 1 Ingress controller 1 Kubernetes API server 1

Contributing factor ?Unreviewed defaults 2 No alert on limit 2 Excessive blast radius 1 No health probes 1 No safeguard on destructive change 1 No headroom 1 Unrepresentative staging 1

Cases

IL-0020 A descheduler evicted pods from cordoned nodes during an EKS upgrade — UK Ministry of Justice, 2025, sev1
Root cause Faulty deployment
IL-0008 A validating webhook with a dead backend stalled a GKE master upgrade — Jetstack, 2019, sev1
Root cause Configuration change
IL-0014 A GKE upgrade that stalled, left zombie nodes and broke an ingress — loveholidays, 2019
Root cause Cloud provider outage

View on the graph