IL-0046 UK Ministry of Justice · 2024-04-15kubernetesawsobservability

Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed

A planned change on the Ministry of Justice Cloud Platform restarted Prometheus. The WAL count was high, so the WAL reload took longer than the startup probe allowed; the report gives the default as 15 minutes. The startup probe failed before the reload finished, the container was restarted, and the next start went the same way. Reverting the planned change to rule it out did not help.

At 15:21 the team decided to raise the startupProbe to 30 minutes and applied it at 15:53. The WAL reload then completed and Prometheus became available, 3h 21m after it was first seen down. Users were updated at 13:00 and 16:00. The incident was closed on 2024-04-18 at 16:13, once the team had found and recorded why the reload took longer.

Severity: –
Scope: Prometheus unavailable; the Cloud Platform lost monitoring and metrics for the period
Users affected: –
Region: –
Time to detect: –
Time to resolve: 3h21m

Symptoms

At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

By 12:52 the team found that Prometheus never finished reloading its write-ahead log (WAL), the on-disk log of recent samples it replays at startup: the pod was restarted before the reload completed. Reverting the planned change at 12:57 to rule it out did not help. At 13:46 the logs showed a startupProbe failed event.

Symptom ? Monitoring outage Probe failures

Root cause

Root cause ? Capacity exhaustion

Component ?Prometheus

Contributing factor ?Unreviewed defaults Unknown limit

Trigger: A planned change restarted Prometheus at a time when the WAL count was high.

Detection

The team saw that the Prometheus pod had restarted several times after the planned change.

Remediation

raised the Prometheus startupProbe threshold to 30 minutes

Action items

Perform planned Prometheus restarts when the WAL count is lower unknown
Set the default CPU and memory requests to meet maximum usage done
Create a test setup that recreates the live WAL count unknown
Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags unknown
Explore storing WAL files remotely in a different location unknown
Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live unknown
Spike into Amazon Managed Prometheus unknown

Lessons

–

Sources

https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2024-04-15-prometheus.html · runbook

Related cases

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024
same root cause · shared Prometheus
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023
same root cause · shared Prometheus
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023
same root cause · shared Prometheus
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023
same root cause · shared Prometheus
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020
same root cause · shared Prometheus