IL-0004 Grafana Labs · 2019-07-19kubernetes
A cluster-wide change had introduced priority classes and made medium the default for pods without one, but the production ingesters had not yet been given the high priority. When a new tenant cluster was created it was scheduled at medium priority, preempted production ingesters, and the replacement ingesters were themselves medium priority, so the preemption cascaded. The cluster had no spare capacity to absorb the new workload. Error-budget alerts paged on-call within four minutes. Scaling the underlying cluster up and letting the ingesters flush restored service in under half an hour, with no data lost because clients replayed their writes.
Ingesters started leaving the ring and remote writes failed, so hosted Prometheus was unavailable for about half an hour. The authenticating proxies in front of it ran out of memory under the retries from customers' Prometheus servers.
Symptom ? Pod evictionService outageOOMKilled
Root cause ? Configuration change
Trigger A new dedicated Cortex cluster was created for a customer without the newly introduced priority classes, so its pods got the default medium priority.
New error-budget-based alerts paged on-call.
Changes to global cluster configuration need more eyes and a written design. A new default must not depend on a setting that is not yet deployed everywhere.
IL-0020 — linked by the editor