IL-0207 GitLab · 2022-11-30kubernetes
On 30 November 2022 GitLab.com went down for all traffic. An engineer had started a routine task from a local development environment, using GitLab's tooling and a well-known open source tool. Through what the review calls an unfortunate series of events and unexpected tool behaviour, the context switched from local development to production during a local deploy or rollback operation, and the change deleted the entire Consul release from GitLab's production Kubernetes cluster.
Consul is a core component of Patroni, which keeps GitLab's Postgres clusters highly available. Without Consul the Patroni clusters became unhealthy and GitLab.com answered with 5XX errors. Engineers were alerted immediately, followed the trail from the Patroni failover alert to the missing Consul servers and found the root cause within minutes. They re-applied the Consul release to production; the Consul cluster came up quickly and the database clusters recovered. The deployment tooling did not make it simple to redeploy the release, and the redeploy needed manual steps.
GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.
Patroni, the high-availability system for GitLab's Postgres clusters, was investigated first and was not responding. Patroni depends on Consul, and the Consul servers were missing. In the Kubernetes cluster the whole Consul release was gone: all Consul pods, services and the other resources of the release had been deleted.
Root cause ? Operator error
GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.
A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default.
Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.
–