IL-0209 GitLab · 2021-08-24kubernetes
During the migration of the GitLab.com web service from VMs to Kubernetes, the target CPU utilisation set on the web pods' autoscaler was too high. As traffic was moved onto the pods, the too-high target kept them from scaling out, and some web requests slowed down. The web latency SLO was breached for about 15 minutes, from 07:30 to 07:45 UTC. The team merged a change (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149) that set a lower, more conservative CPU target. The incident review sections of the issue were left unfilled.
The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.
Symptom ? High latencyAutoscaler misbehaviour
Root cause ? Configuration change
The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.
When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.
–