IL-0205 GitLab · 2021-06-21kubernetesgcp
GitLab.com runs its Kubernetes services on GKE in us-east1, on node pools backed by SSD persistent disks. Google Cloud ran short of SSD stock in that region, so neither the cluster autoscaler nor the CI runners could create machines. The shortage started to hurt the API service on 2021-06-21, when it needed to scale up with user traffic and no new nodes came up; for about 146 minutes the API was degraded with slow responses.
Google Cloud support said the shortage did not affect pd-standard disks, so the team added a node pool with pd-standard disks and, when it did not scale up by itself, scaled it by hand across all three zones to relieve the API. Auto-Deploy stayed blocked for about 1.5 days. Moving the other workloads to new pd-standard node pools exposed a cluster autoscaler bug: when one node pool that matches a pod's selector cannot scale, the autoscaler does not try another node pool whose labels also match. Google listed this as a known issue with no date for a fix, so the team changed node pools and nodeSelectors for the remaining workloads. The first canary deploy completed at 16:28 and the first production deploy since the start of the incident at 19:21 UTC on 2021-06-22.
After about a day running mainly on the pd-standard node pools with no problems, the team removed the SSD-backed node pools. The review records a time to detection of 22 hours and states that no GitLab change triggered the incident.
At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.
The first sign had come the day before: at 10:30 UTC on 2021-06-20 the cluster autoscaler first failed to add nodes to the production cluster for a Sidekiq workload, and at 05:51 on 2021-06-21 the CI runners began failing to start new machines (tracked separately in issue #4937). The cluster autoscaler errors said that some node pools were in a backoff state but not why. At the instance-group level the only error was ZONE_RESOURCE_POOL_EXHAUSTED, which did not say which resource was exhausted. At 08:19 Google Support told GitLab that Google was low on SSD stock.
Once the API had recovered, Auto-Deploy stayed blocked, because new pods in staging, canary and production all needed new nodes. A node pool with pd-standard disks added at 09:55 did not scale up on its own, and at 18:16 testing in the canary stage showed that the cluster autoscaler did not work with the new node pools, so production deployments were stopped.
Symptom ? Scale-up failureHigh latencyPods PendingDeploys failing
Root cause ? Cloud provider outage
There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.
A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support.
The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.
same root cause · shared Node pool, Cluster autoscaler
same root cause · shared Node pool, Cluster autoscaler