IL-0096 2i2c · 2022-11-14kubernetesgcp
2i2c runs the LIS JupyterHub on a regional GKE cluster in europe-west2 (London). To get the best performance from its NFS Filestore, the nodes were restricted to the B zone, where the Filestore is. When Google's europe-west2-b zone ran out of resources at 10 AM UK time, the cluster autoscaler could not add nodes, new user server pods could not be scheduled, and the hub gave up on each server after 600 seconds.
For a long period one engineer handled the incident alone as incident commander, communicator and debugger, and pages did not reach the other response engineers. The error message pointed toward quotas, so time went into raising the SSD and CPU quotas, resizing the node pool by hand and restarting the hub, none of which helped. The ZONE_RESOURCE_POOL_EXHAUSTED error in the logs showed that Google had no more resources in the requested zone.
The affected node pool was edited in the GCP console to allow nodes in the A and C zones of the region as well, and new nodes came up in europe-west2-c. The report calls this a temporary fix, because the NFS Filestore is now used across zones, which costs performance. Impact lasted from 02:00 to 08:30 Pacific, 6 hours 30 minutes.
A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.
The pod events read "Warning FailedScheduling 3m29s http://gke.io/optimize-utilization-scheduler 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling." and "Normal NotTriggerScaleUp 3m27s cluster-autoscaler pod didn't trigger scale-up: 1 node(s) didn't match Pod's node affinity/selector, 1 in backoff after failed scale-up". The hub log showed "sgibson91's server failed to start in 600 seconds, giving up", "3 consecutive spawns failed. Hub will exit if failure count reaches 5 before succeeding" and "asyncio.exceptions.TimeoutError: pod lis/jupyter-sgibson91 did not start in 600 seconds!".
Because "backoff after failed scale-up" usually means a quota problem, the engineer first doubled the Persistent Disk SSD quota, which was shown in red, and later doubled the CPU quota to 48; neither helped. A 100GB persistent volume claim belonging to the Prometheus server turned up but did not explain the failure. Resizing the nb-user node pool to one node by hand completed in the GCP console, yet the node count stayed at 0 and kubectl get nodes showed no new node. Deleting the hub pod did not help either.
The logs turned out to contain ZONE_RESOURCE_POOL_EXHAUSTED errors. Another engineer found that the first one had appeared at 10 AM UK time, three hours and forty minutes before the incident was opened.
Root cause ? Cloud provider outage
A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.
A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity.
The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.
same root cause · shared Cluster autoscaler, Node pool
same root cause · shared Cluster autoscaler, Node pool