IL-0154 DigitalOcean · 2024-09-25kubernetesnetworking
On September 25, 2024, hardware errors made most of the line cards on one of DigitalOcean's core routers in SFO3 reboot at once. Traffic capacity between the datacenter and the network backbone fell by half and inter-regional traffic was interrupted. The line cards came back within three minutes, routing sessions re-established, and traffic on the router was restored at 22:30 UTC.
A number of machines in the DOKS fleet stayed unhealthy because their guest networking did not recover from the fault. Their clusters' API servers degraded and nodes went NotReady, and engineers had to remediate the nodes by hand across the entire SFO3 DOKS fleet. All affected nodes were recycled and the clusters were operational at 01:40 UTC on September 26, so DOKS users were affected for about three hours longer than users of other services.
From 22:25 to 22:50 UTC every DigitalOcean service in the SFO3 region had network connectivity problems, as traffic between the datacenter and the backbone dropped. Traffic through the router was back by 22:30 and SFO3's control plane systems had reconnected by 22:50, and most services recovered then.
Managed Kubernetes (DOKS) clusters in SFO3 did not recover. At 23:07 UTC the DOKS API servers were degraded, and customers had trouble connecting to their clusters' control plane and reaching other services from inside their clusters. At 23:59 some DOKS clusters could no longer be scraped, and several nodes were found in the NotReady state: the guest networking on those machines had not recovered after the network fault. DigitalOcean's first status notice, at 23:34, was about DOKS connectivity; the earlier disruption of every service in the region from 22:25 to 22:50 was identified only later, during that investigation.
Symptom ? Node NotReadyControl plane unresponsiveIntermittent connection failures
Root cause ? DNS or network fault
–
A short network fault can leave Kubernetes nodes broken long after the network itself is healthy, when the guest networking on the machines does not recover by itself. Checking node health and the API servers after the network returns, and having automated remediation that is allowed to replace many nodes at once, shortens that tail.
–