IL-0095 2i2c · 2025-08-12kubernetes
2i2c runs JupyterHub deployments, including the LEAP hub. An engineer deploying a new hub hit a bug in 2i2c's NFS deployment process, and while working around it deleted the prod namespace. The kubeconfig context in use was the LEAP cluster, not the new one, and the engineer had no contextual clue that they had switched onto the LEAP hub. Every pod in LEAP's prod namespace was terminated and the hub was down for about an hour. The report names the NFS deployment bug as the root cause.
A second engineer joined early to handle communication with the LEAP community, and the People Operations Lead later took over the communications part of the incident commander role. The engineer knew the fix was to redeploy prod but was not confident it carried no risk of losing persistent data, and reaching the tech lead in another time zone took time; the tech lead confirmed the plan at 5:43 PM. The last remaining user pod in prod was deleted so the namespace deletion completed. The nfs-share-creator job was blocking the deployment and was killed at 5:48 PM, prod was redeployed with the updated NFS server IP address at 5:51 PM, and the prod ingress was deleted at 5:55 PM so that users could not start servers mid-deployment. The hub was declared online at 6:08 PM and the incident resolved at 6:11 PM. Redeployment took longer because of a circular dependency between deploying jupyterhub-home-nfs and putting the NFS server IP into the PVCs.
No data was lost. 2i2c provisions persistent storage separately with Terraform, so deleting Kubernetes resources did not remove user data.
At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.
At 5:02 PM the engineer confirmed the cluster was LEAP, with no pods left in prod. Every pod in the prod namespace had been terminated and the hub was entirely inaccessible. One user pod remained, and it kept the namespace deletion from finishing.
The uptime check on leap.2i2c.cloud failed at 5:32 PM, half an hour after the engineer had declared the incident. At 5:37 PM the first user reports of 404 errors when opening the hub arrived.
After the hub was back, at 8:16 PM, Sarika de Bruyn reported that some users had trouble logging in or saw kernel disconnections. The engineer saw heavy Dask workloads starting, with up to 200 Dask nodes being provisioned, and then saw the load settle.
Symptom ? Resources deletedService outage
Root cause ? Operator error
The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.
Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.
same root cause · shared Ingress controller
same root cause · shared Persistent volumes and stateful sets
same root cause · shared Ingress controller
same root cause · shared Ingress controller