IL-0176 LINE · 2020-07-10 (published)kubernetesobservability
The cluster ran Kubernetes 1.13.5 with 235 nodes, 7 control plane nodes and 5 etcd nodes plus worker nodes, all of them VMs. Prometheus 2.14.0 ran with storage.tsdb.retention.size=25GB on a host with 4 vCPUs, 8GB of memory and a 100GB SSD. Exit code 137 means the process received SIGKILL. The author ruled out a manual kill, since only the author had root on the host, and traced through the kubelet and cAdvisor code that kubelet only reports OOM events that cAdvisor reads from /dev/kmsg; so the kernel OOM killer, not kubelet, had killed the container. Prometheus loaded more than two million series within minutes of starting, which needed more memory than the host had.
The author then looked at why the kernel chose Prometheus. The OOM killer in the kernel in use (3.10.0-957.el7) scores processes by memory use adjusted by oom_score_adj, and the Prometheus container's oom_score_adj was 999. A SystemTap script watching writes to /proc/<pid>/oom_score_adj showed no process changing it at run time. The containerd and kubelet code showed that kubelet sets the value when the container is created, from the pod's QoS class: -998 for Guaranteed, 1000 for BestEffort, and for Burstable 1000 minus 1000 times the memory request divided by node memory. In the Prometheus pod only the configmap-reload container had requests and limits (memory 50Mi and 100Mi, CPU 100m and 200m); the Prometheus container had none. The pod was therefore Burstable, and the Prometheus container got 999, which made it the first process to be killed.
The author names two causes: wrong resource requests and limits on the Prometheus pod, which gave the container a high oom_score_adj, and no advance estimate of how much memory Prometheus needs in a large cluster. Removing most of the unnecessary settings from the Prometheus ConfigMap stopped the crashes.
One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.
kubectl describe showed the Prometheus container (prom/Prometheus:v2.14.0) Waiting with Reason CrashLoopBackOff, its last state Terminated with Reason Error and Exit Code 137, while the configmap-reload sidecar was running normally. Because Prometheus had crashed on full disks before, the author checked storage first: the 30Gi PVC was only 35% used (11G of 32G). The Prometheus log showed no error; its last lines were the start of TSDB and the WAL replay ("replaying WAL, this may take awhile", "WAL segment loaded" segment=0 maxSegment=12).
The kubelet log on the host showed the readiness probe failing with "HTTP probe failed with statuscode: 503", then "Got sys oom event from cadvisor", then the liveness probe failing with "read: connection reset by peer" and a PLEG ContainerDied event. Restarting the container with docker restart reproduced it: Prometheus used almost all of the host's memory and the kernel killed it.
The Grafana dashboard for Prometheus showed the number of time series jumping to 2,162,256 within five minutes of start. A memory calculator put that at an estimated 5.7GB, on a host with 8GB of memory and swap disabled.
Symptom ? Container crash loopOOMKilledMonitoring outageProbe failures
Root cause ? Capacity exhaustion
The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.
Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared kubelet
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared kubelet
same root cause · shared Linux kernel