IL-0016 UK Ministry of Justice · 2023-07-25kubernetesawsobservability
The Prometheus instance was sized for an earlier number of namespaces and rules and had outgrown its node. Node memory reached about 89 percent, the pod was OOM-killed, the node went NotReady and Prometheus was rescheduled. The team drained the monitoring node, moved to a larger instance type, and raised the container memory limit in two steps. Repair took under two hours. The follow-up was an alert on node memory and on pods using most of a node's memory.
The external health check on the live Prometheus failed and paged. The Prometheus pod had been killed for memory and its node went NotReady.
Symptom ? Monitoring outageOOMKilledNode NotReady
Root cause ? Capacity exhaustion
A Pingdom health check failed and raised a PagerDuty high-priority alert.
The report states no lessons beyond the alerting action.
IL-0013 Connection resets from a full conntrack table on GKE nodessame root cause · shared Node pool
IL-0015 A GKE pod address range ran out at 256 nodessame root cause · shared Node pool
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memorysame root cause · shared Container memory limit
IL-0001 — linked by the editor