IL-0153 DigitalOcean · 2023-10-08kubernetesnetworking
The Debian image used for DOKS worker nodes on cluster versions 1.27.6-do.0 and 1.28.2-do.0 had a systemd timer enabled that ran an apt upgrade of all packages every day at 06:00 UTC plus up to 60 minutes. The upgrade pulled in unsupported versions of system-critical packages, including systemd and the kernel. The kernel upgrade could take longer than 15 minutes, and systemd then aborted the upgrade ungracefully, which in most cases left the node's public and private networking broken. The node went NotReady and stayed that way until it was replaced or rebooted.
The incident repeated on October 8, 9 and 10, 2023, across the AMS3, BLR1, FRA1, LON1, NYC1, NYC3, SFO2, SFO3, SGP1, SYD1 and TOR1 regions. Investigation was slow because the affected nodes had no working network and engineers' access to worker nodes is restricted. After cron jobs were ruled out, the team found the timer on October 10 at 13:08 UTC, ran it by hand on a test cluster, and saw a long upgrade of systemd and the kernel begin; the effect of the 15-minute timeout could not be fully validated, but the evidence was judged sufficient. A DaemonSet that disables the timer on every worker node was released to all affected clusters on October 11 at 03:08 UTC, and no failures occurred at 06:00 UTC that day.
Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.
On an affected node, processes could not reach anything over either the VPC or the public network. Traces across the worker nodes and their hypervisors showed reply packets arriving at the machine but never reaching the processes that had sent the requests, including kubelet and workload pods, so the network data path was ruled out and the fault was placed in the guest operating system.
The broken networking also kept engineers out. DOKS worker nodes allow DigitalOcean engineers only limited access for security reasons, so SSH and console access had to be arranged while the nodes were healthy, and live debugging on an affected node was possible only on the third occurrence. Many clusters appeared to heal on their own or were fixed by customers rebooting nodes, and attempts to reproduce the failure failed until the daily 06:00 UTC pattern pointed at a scheduled job on the nodes.
Symptom ? Node NotReadyService outage
Root cause ? Configuration change
Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.
An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs.
When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.
same root cause · shared kubelet
same root cause · shared kubelet