IL-0208 GitLab · 2022-03-31kubernetesgcpnetworking
The change was deployed in turn to a canary environment, gprd-cny, and to the clusters gprd-us-east1-b, gprd-us-east1-c and gprd-us-east1-d. Merge request !1643 bumped the GitLab chart, and the new chart renamed ports on Kubernetes Services and Deployments for the Prometheus service and pod monitors. Applying that rename made the GCP load balancer reconfigure, and the new load balancer configuration needed some minutes to apply fully. During those minutes requests were slow and some timed out, which users saw as slowness and 50x errors. GitLab classifies the root cause as a configuration change.
The EOC drained the canary at 08:05 and re-enabled it at 08:25, and an engineer cancelled the deployment jobs for clusters c and d at 08:15 before they started. The pipeline nevertheless began deploying to gprd-us-east1-d at 09:04. At 09:31 the team raised the minimum replica count above 100. The change was then rolled out to gprd-us-east1-c in a controlled way: the cluster was drained in HAProxy at 09:53, the change was deployed at 09:57 and the cluster was re-enabled at 10:07. At 10:19 the minimum replica increase was reverted. GitLab sums up the mitigation as draining the clusters and running a controlled rollout of the change, and estimates that 173896 unique IP addresses received a 50x response between 07:53 and 10:10 UTC.
At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.
The errors started four minutes after a k8s-workloads/gitlab-com pipeline began deploying merge request !1643 to the canary environment gprd-cny at 07:49. The change was a GitLab chart bump that renamed ports on Kubernetes Services and Deployments, a rename related to the Prometheus service and pod monitors. The engineer who was deploying the change noticed the errors and tied them to it.
The impact came in five separate windows, 78 minutes in total: 07:53 to 08:02, 08:05 to 08:38, 08:53 to 09:01, 09:07 to 09:14 and 09:55 to 10:16 UTC. Over the same morning the change reached the gprd-us-east1-b cluster at 08:03, started deploying to the gprd-us-east1-d cluster at 09:04 even though the deployment jobs for clusters c and d had been cancelled at 08:15, and the EOC started deploying it to the gprd-us-east1-c cluster at 09:57.
Declaring the incident was slowed down as well: at 07:58 the Woodhouse slash command did not respond, and the EOC declared the incident in Slack at 08:03.
Root cause ? Configuration change
A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.
Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes.
GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.
same root cause · shared Cloud load balancer, HAProxy
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer