Incident Library?

Incident Library?

Symptom ?

High latency 지연 시간 증가 · `high-latency`

3 cases · 2019 to 2024

Root cause ?Software defect 3

Component ?Cluster autoscaler 1 Configuration templating 1 Container CPU limit 1 Application database 1 HAProxy 1 Ingress controller 1 kubelet 1 Linux kernel 1 Model hub 1

Contributing factor ?Unverified assumption 1 Ignored early signal 1 Unpatched known bug 1 Unknown limit 1 Monitoring blind spot 1 No headroom 1 Overlapping failures 1 Retry storm 1 Unrepresentative staging 1

Cases

IL-0024 Hub outage from expensive cache queries and uncancelled requests — Hugging Face, 2024, sev1
Root cause Software defect
IL-0012 CPU limits and a kernel scheduler bug stalled multi-threaded services — Omio, 2020, sev3
Root cause Software defect
IL-0011 Sorted HAProxy backends sent every request to the same first pod — Skyscanner, 2019, sev2
Root cause Software defect

View on the graph