IL-0094 2i2c · 2020-10-28kubernetesgcp
2i2c runs a JupyterHub for WER on a GKE cluster. The core pods did not have appropriate resource guarantees, and there was no dedicated pool for them, so the students' pods ran on the same nodes. Each student had a memory limit of 2G and a guarantee of only 512M, the 4x overcommit 2i2c often uses, but earlier tests had shown that students almost always use just under 1G, so the overcommit should have been only 2x. The students filled the CPU and RAM of the nodes, everything was starved of resources, and the hub froze.
The team removed memory overcommit for the WER students, which was also expected to spawn another node and relieve the existing ones, and raised resource guarantees for all core pods so they could keep working on full nodes. That change restarted the pods and moved some to a new node, and things returned to normal; the issue was closed at 09:46 PM.
At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.
At 09:21 PM the node resource use made the exhaustion clear. kubectl top node showed gke-low-touch-hubs-cluster-core-pool-b7edea69-gwrg at 1944m CPU (100%) and 10432Mi memory (98%), and gke-low-touch-hubs-cluster-core-pool-b7edea69-00sc at 220m (11%) and 6151Mi (58%). The cluster had only core nodes and no separate user nodes, so the suspicion was that the student pods used just enough resources to starve the core pods.
Symptom ? Service outageSustained high node CPU
Root cause ? Capacity exhaustion
WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).
Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.
same root cause · shared Container memory limit, Node pool
same root cause · shared Container memory limit, Node pool
same root cause · shared Container memory limit, Node pool
same root cause · shared Container memory limit, Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Container memory limit