Incident Library?

Incident Library?

Symptom ?

OOMKilled OOMKilled · `oom-killed`

4 cases · 2019 to 2023

Root cause ?Capacity exhaustion 2 Configuration change 1 Software defect 1

Component ?Container memory limit 2 Configuration templating 1 CoreDNS 1 Client-side DNS resolver and cache 1 HAProxy 1 Ingress controller 1 Node pool 1 Pod priority and preemption 1 Prometheus 1

Contributing factor ?Retry storm 2 Unverified assumption 1 Ignored early signal 1 Unknown limit 1 Monitoring blind spot 1 No alert on limit 1 No caching 1 No headroom 1 Missed in review 1 Unrepresentative staging 1

Cases

IL-0016 Prometheus on the live cluster killed by memory exhaustion — UK Ministry of Justice, 2023, sev3
Root cause Capacity exhaustion
IL-0004 Pod priority defaults preempted production ingesters — Grafana Labs, 2019, sev2
Root cause Configuration change
IL-0011 Sorted HAProxy backends sent every request to the same first pod — Skyscanner, 2019, sev2
Root cause Software defect
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019
Root cause Capacity exhaustion

View on the graph