Incident Library?

Incident Library?

Contributing factor ?

Unrepresentative staging 스테이징 불일치 · `staging-not-representative`

3 cases · 2019 to 2024

Root cause ?Cloud provider outage 1 Configuration change 1 Software defect 1

Component ?Ingress controller 2 Cluster autoscaler 1 Configuration templating 1 Client-side DNS resolver and cache 1 GKE control plane 1 HAProxy 1 Horizontal pod autoscaler 1 Kubernetes API server 1 Node pool 1 Telemetry agent 1

Symptom ?Service outage 2 Autoscaler misbehaviour 1 Control plane unresponsive 1 DNS resolution failures 1 High latency 1 Intermittent connection failures 1 OOMKilled 1 Load imbalance 1 Upgrade stall 1

Cases

IL-0005 Telemetry rollout overloaded the control planes and broke service discovery — OpenAI, 2024, sev1
Root cause Configuration change
IL-0011 Sorted HAProxy backends sent every request to the same first pod — Skyscanner, 2019, sev2
Root cause Software defect
IL-0014 A GKE upgrade that stalled, left zombie nodes and broke an ingress — loveholidays, 2019
Root cause Cloud provider outage

View on the graph