IL-0099 2i2c · 2025-05-12kubernetes
2i2c runs a JupyterHub for the Earthscope community. During a workshop, users started servers from a legacy profile option that guaranteed each server 8 GB of memory on a 32 GB node but set no memory limit. When users kept using more than their guarantee, the node's memory was heavily overcommitted, users' pods evicted each other, kernels were killed and server starts took more than 15 minutes.
The first response on the evening of 12 May was a helm rollout and hub redeploy, which helped only partly; the instructor switched to a local Python environment for the rest of the day. On 13 May at 14:30 the cause was identified as the legacy profile option. The engineers did not want to change the profile options so soon before that day's workshop, so they made the 'medium' profile available so more users could share a machine, raised its guarantee so the overcommitment was smaller, and pre-warmed the cluster with 15 nodes for the 30 users. The community confirmed at 19:28 that the workshop was running smoothly, and the minimum node count was set back to zero at 19:31.
On 14 May, before the next workshop, the engineers changed the community's profile options for small nodes so that the memory guarantee equals the memory limit. The community confirmed that workshop went well.
On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.
A helm rollout and hub redeploy at 21:05 seemed to help, and the community could launch servers again, but things still hung. At 22:00 an engineer reported the problem was still there, with the error `tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist`.
A second engineer went through the saved logs, found nothing obviously wrong with the hub, and concluded it was resource exhaustion. A Grafana panel showed high CPU usage on the core node where the hub was running.
Symptom ? Pod eviction
Root cause ? Capacity exhaustion
A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.
A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment.
For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.
same root cause · shared Node pool, Container memory limit
same root cause · shared Node pool, Container memory limit
same root cause · shared Node pool, Container memory limit
same root cause · shared Node pool, Container memory limit
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Container memory limit