IL-0070 2i2c · 2023-02-01kubernetesobservability
2i2c runs JupyterHub with dask-gateway for the LEAP community. On 1 February 2023 one user's dask-gateway workload scaled the cluster by about 100 nodes in about a minute. Prometheus ran on one of the four core nodes next to the hub and already requested most of that node's memory. The incident report's author believes that various critical pods on the core nodes were evicted when Prometheus started scraping metrics exporters on about 200 nodes. The hub pod restarted.
The notes written during the incident give two possible reasons for the hub restart: eviction under memory pressure on the node shared with Prometheus, or a hub that stopped responding, perhaps starved of CPU with a 10m request, and was restarted after its liveness probe failed 30 times in a row. The report adds that the dask scheduler pod would likely, though not certainly, also hit resource limits with that many workers. No single cause was proven, partly because the events had expired before an OOM kill could be checked.
The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.
Within about a minute roughly 100 nodes were added, probably along with 100 or more dask-worker pods. The dask workers' scheduling events read: `0/126 nodes are available: 14 node(s) didn't match Pod's node affinity/selector, 23 Insufficient cpu, 24 Insufficient memory, 6 node(s) had untolerated taint {node.kubernetes.io/network-unavailable: }, 9 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 96 node(s) had untolerated taint {node.kubernetes.io/not-ready: }.`
The core node with the hub also ran support-prometheus-server, which requested 3 CPU and 20Gi of memory; the node had 88% of its CPU and 93% of its memory requested, and memory limits at 124%. The engineer could not confirm an OOM kill with `kubectl get event -A | grep OOM`, because the API server keeps events for only 60 minutes. Grafana showed the hub's memory steady, so a second theory was that the hub became unresponsive and its liveness probe restarted it; the hub and proxy pods requested only 10m of CPU.
Symptom ? Pod evictionPods Pending
Root cause ? Capacity exhaustion
A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.
–
Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.
same root cause · shared Node pool, Prometheus
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Node pool
same root cause · shared Prometheus
same root cause · shared Node pool
same root cause · shared Node pool