IL-0086 CircleCI · 2021-03-11kubernetesnetworking
On 11 March 2021 CircleCI deployed an update to a test Kubernetes cluster. The deployment removed Traefik and Calico resources from both the test cluster and CircleCI's primary production cluster. Without Traefik, the CircleCI UI and runner.circleci.com were unreachable; the API has its own ingress point and was not affected.
Because the connection to the test deployment was not obvious, engineers first investigated DNS and AWS causes. A deeper look showed the Traefik problem, and Traefik was restored quickly, but that did not fix everything. Engineers then found that the link between Calico and Traefik was the root cause; once Calico was restored, pods in the production cluster went back to serving customer requests. The outage lasted 1 hour 54 minutes.
At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.
Later in the incident CircleCI found that jobs on CircleCI Runner were not running either, while all builds outside Runner ran as expected.
Nothing pointed at the test cluster, so engineers first chased DNS and AWS-related leads, which turned out to be red herrings. When the fix was in, CircleCI said that because the problem was DNS-related some customers might need to clear their browser's DNS cache, and some locations might see the change a little later.
Symptom ? Service outageResources deleted
Root cause ? Faulty deployment
CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.
A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again.
Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.
same root cause · shared Ingress controller
same root cause · shared Ingress controller