Incident Library?

Incident Library?

Symptom ?

Monitoring outage 모니터링 중단 · `monitoring-down`

2 cases · 2019 to 2023

Root cause ?Capacity exhaustion 2

Component ?Container memory limit 2 CoreDNS 1 Client-side DNS resolver and cache 1 Node pool 1 Prometheus 1

Contributing factor ?Unknown limit 1 Monitoring blind spot 1 No alert on limit 1 No caching 1 Retry storm 1

Cases

IL-0016 Prometheus on the live cluster killed by memory exhaustion — UK Ministry of Justice, 2023, sev3
Root cause Capacity exhaustion
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019
Root cause Capacity exhaustion

View on the graph