IL-0200 GitHub · 2026-03-12kubernetes
GitHub was carrying out a planned infrastructure upgrade in which Kubernetes nodes were replaced. A configuration issue meant the newly provisioned nodes came up running an incompatible version of etcd, the key-value store that holds Kubernetes cluster state and only works while its members agree with each other (consensus). The incompatible members disrupted consensus in several production clusters, and between 01:36 and 08:11 UTC GitHub.com served intermittent 5XX errors on Git operations and web requests.
Once engineers identified the incompatible nodes, they removed them and cluster consensus was restored. Deployments stayed paused until a validation deploy had confirmed that all systems were healthy, and then normal operations resumed. GitHub's follow-up work is to enforce version compatibility in code whenever nodes are replaced, to detect split-brain conditions (cluster members that no longer agree on state) earlier, and to make the recovery tooling restore clusters faster.
From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.
Users hit intermittent 5XX errors on git push, git clone and page loads. At 02:13 UTC the status page reported that error rates were decreasing, and at 02:44 UTC that the root cause had been identified while errors continued at a lower rate. Deployments were paused for the whole incident, which lasted until 08:11 UTC.
Symptom ? High error rate
Root cause ? Configuration change
The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.
–
same root cause · shared etcd