IL-0044 Saifeddine Rajhi · 2024-06-14kubernetesobservability
When Prometheus restarts it rebuilds its in-memory state by replaying the write-ahead log, the on-disk record of recent samples. In this cluster the replay needed two to three times the memory Prometheus used while running, more than 50Gi against about 30Gi in steady state, so each start ended in an OOMKill before Prometheus became ready. The problem showed up while the team was upgrading the kube-prometheus-stack chart.
The author links the problem to an upstream GitHub issue open since 2020 and to two common situations: scraping too many series, and running Prometheus close to its memory limit so that any replay overhead pushes it over. Both come down to label cardinality. With Prometheus down, PromQL could not be used to find the expensive metrics, so the author ran promtool tsdb analyze inside the Prometheus pod with kubectl exec; it pointed to high-cardinality metrics such as haproxy_server_http_responses_total. Reducing that metric, giving Prometheus more memory, or moving it to a node group with more memory are the fixes the author proposes; the write-up does not say which one ended the crash loop.
After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.
The Prometheus log contained: level=info component=tsdb msg="Deletion of corrupted mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 946594555 ...". kubectl describe pod showed the container in State: Running but with Last State: Terminated and Reason: OOMKilled.
Prometheus was being killed while replaying its write-ahead log (WAL) at startup. In normal operation it used around 30Gi of memory, but the replay needed more than 50Gi. Raising the memory limit alone was not seen as a fix, because the extra memory was needed only during the replay phase.
Symptom ? OOMKilledProbe failuresMonitoring outage
Root cause ? Capacity exhaustion
The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.
Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.
same root cause · shared Prometheus, Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Prometheus
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Prometheus
same root cause · shared Container memory limit
same root cause · shared Prometheus
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Prometheus
same root cause · shared Container memory limit
same root cause · shared Container memory limit