IL-0045 UK Ministry of Justice · 2023-11-01kubernetesawsobservability
Prometheus on the Cloud Platform's live cluster restarted several times, and each time the platform lost monitoring for a while, which left gaps in the metrics. Before the restarts, Prometheus logged many rule evaluation failures and its readiness probe failed.
After going through the logs, the team concluded that one possible root cause was the readiness probe failing before Prometheus restarted, and raised the readiness probe timeout from 3 to 6 seconds; the incident was then marked repaired and resolved. The report does not establish why the readiness probe failed and lists open questions in its review actions: whether data ingestion is too big or too long, whether some queries make Prometheus work so hard that it stops answering the probe, whether other services probing Prometheus trigger the restart, and whether regular Velero backups disturb EBS reads and writes.
At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.
The Prometheus pod had restarted and the prometheus container was starting up again. Prometheus logs showed numerous "Evaluation rule failed" messages, and events in the monitoring namespace recorded "Readiness Probe failed" for Prometheus. While Prometheus was down, the Cloud Platform had no monitoring and metrics went missing.
At 12:35 the team switched Prometheus to the debug log level to understand the problem.
Symptom ? Monitoring outageProbe failures
Root cause ? Not stated
A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.
–
–