Incident Library?

Incident Library?

Contributing factor ?

Retry storm 재시도 폭주 · `retry-storm`

3 cases · 2019 to 2024

Root cause ?Capacity exhaustion 1 Configuration change 1 Software defect 1

Component ?Cluster autoscaler 1 CoreDNS 1 Application database 1 Client-side DNS resolver and cache 1 Container memory limit 1 Model hub 1 Pod priority and preemption 1

Symptom ?OOMKilled 2 Service outage 2 Database overload 1 DNS resolution failures 1 High error rate 1 High latency 1 Monitoring outage 1 Pod eviction 1

Cases

IL-0024 Hub outage from expensive cache queries and uncancelled requests — Hugging Face, 2024, sev1
Root cause Software defect
IL-0004 Pod priority defaults preempted production ingesters — Grafana Labs, 2019, sev2
Root cause Configuration change
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019
Root cause Capacity exhaustion

View on the graph