IL-0206 GitLab · 2022-04-25kubernetesgcp
Every GitLab.com pod on GKE runs a Consul agent, which the Rails application uses to discover which PostgreSQL databases are healthy, up-to-date replicas. Under certain circumstances the agent's memory demand spikes, and with a pod memory limit of 500MB the agent was killed again and again. Without a working Consul agent, the Rails pods (web, api, git and websockets) kept failing to discover replicas and fell back to sending all their queries to the last known primary database. The primary saturated, and Sidekiq processing and the whole site slowed down intermittently on 2022-04-25 and 2022-04-26.
During the first incident the team worked from a different hypothesis: that a pgbouncer pool reduction made on 2022-04-21 had saturated connections. The plan was to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database, and to consider rolling back the CI feature flag from 50%. Saturation and slowness dropped and the incident was marked mitigated, but the elevated error rate in the early hours of 2022-04-26 turned out to be related, and further investigation tied both events to the Consul OOM kills.
As a temporary fix, the pod memory limits were raised from 500MB to 1GB so the Consul agents keep running through a spike. The team planned to watch the agents with the new limit and to find out whether the spikes are expected Consul behaviour or come from GitLab's configuration.
On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.
The first suspect was a pgbouncer change made on 2022-04-21 that had reduced connection pool sizes. At 16:23 the working hypothesis was that this reduction had saturated database connections; the team also looked at other recent deploys and at a feature flag rollout at 50% for CI phase 4. The Sidekiq queue had been in a bad state at 15:33 but had recovered by 16:03, and one engineer pointed out that the saturation metric was slightly incorrect, so saturation might be lower than it looked.
The incident was marked mitigated at 17:08 because saturation and site slowness had dropped, and changes were applied at 17:26 (the timeline does not list them). At 01:21 UTC on 2022-04-26 monitoring detected an elevated error rate, first declared as a separate incident and later tied to the same cause. Further investigation found OOM kills on the GKE nodes: under certain circumstances the Consul agent in each pod had a spike in memory demand, and when the memory was not available it was killed rapidly and repeatedly.
Symptom ? High latencyDatabase overloadOOMKilledHigh error rate
Root cause ? Capacity exhaustion
An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.
When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.
same root cause · shared Container memory limit, Application database
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Container memory limit