IL-0152 DigitalOcean · 2024-01-23kubernetes
In FRA1, the control planes of a subset of DOKS clusters ran on an internal infrastructure cluster. During a maintenance operation an incorrect configuration was inadvertently applied to that cluster at 12:21 UTC. It set off an automatic deletion process on the compute resources hosting the control planes, and by 12:30 all DOKS control planes on that infrastructure cluster had been removed, with loss of control plane data.
The configuration was identified as the cause at 12:36. Recovery restored each cluster's etcd from its latest backup; every DOKS etcd cluster is backed up twice a day on a randomized schedule per cluster, so in the worst case a cluster's control plane data was up to 12 hours old. After etcd, the stateless control plane components were restored. Unexpected problems with the recovery mechanism from 13:00 needed extra patching and were fully addressed by 15:00. All affected highly available DOKS clusters were recovered by 20:31 and all affected clusters by 21:21. Customers might need to reconfigure some Kubernetes resources, and support contacted them directly.
At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.
At 15:12 UTC the status page said 200 clusters were affected and still inaccessible. At 18:44 about 10% of clusters had their access restored, and an additional small bug had been found and fixed. Workloads already running on the clusters' nodes kept running until recovery was complete.
Root cause ? Configuration change
Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.
When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.
same root cause · shared etcd
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server