IL-0198 Datadog · 2023-03-08kubernetesnetworkingawsgcpazure
In December 2020, a change in systemd v248 made systemd-networkd remove every IP rule it does not know about when it starts; v249 added the ManageForeignRoutingPolicyRules setting to turn that off, and both changes were backported to v248 and v247. Ubuntu 22.04 ships systemd v249 with the default that manages foreign rules, while Ubuntu 20.04 uses v245 without either change. Datadog had tested 22.04 from June 2022 and rolled it out progressively from November 2022, reaching more than 90 percent of the fleet by 8 March. The new behaviour had never shown, because systemd-networkd only started when a host was created, before any routing rules existed.
Pods at Datadog get IP addresses from the cloud network, and Cilium makes this work by installing host routes and rules. On AWS (ipam=eni) and Azure (ipam=azure) pods use an extra network interface, so Cilium adds source routing rules per pod. On all three providers Cilium also moves the default "from all lookup local" rule from priority 0 to 100, so that traffic leaving its L7 proxy can be routed correctly. When the patched systemd restarted systemd-networkd, it deleted these rules and did not put back the priority-0 local rule. Traffic to the node's own IP was no longer treated as local, so every affected node lost host connectivity on all providers, and pods also lost connectivity on AWS and Azure.
The nodes kept Ubuntu's default unattended-upgrades settings: download package updates twice a day with up to 12 hours of random delay, and install security updates once a day at 06:00 UTC with up to 60 minutes of delay. Datadog does not depend on these upgrades and patches its fleet by replacing nodes progressively, cluster by cluster and region by region, with bake times of a week to a month, but unattended upgrades had never been disabled. They applied the patch everywhere within the same hour. Nodes on Ubuntu 20.04, and nodes that had not yet downloaded the new package (for example Google Cloud nodes whose update check ran before about 18:50 UTC on 7 March), were not affected. In total more than 60 percent of instances were disconnected.
Recovery depended on the provider. On Google Cloud and Azure, thousands of instances were restarted through the cloud APIs and kept their local disks. On AWS, automatic replacement of tens of thousands of instances destroyed local-disk data that many data stores, caches and coordination systems relied on, so that data had to be rebuilt through replication, reprocessing or restore from backup. Datadog understood the cause within the first two to three hours; the recovery took longer and is described in a separate post.
Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.
On Google Cloud and Azure the instances kept running without network. Packets sent per second across the EU1 region fell from hundreds of millions to almost zero between 06:00 and 07:00, and later rose to twice the usual rate while the backlog was processed. The providers did not replace these instances, because they only auto-repair guest operating system failures when HTTP health checks are configured, and Datadog relies on the Kubernetes control plane instead.
On AWS, instances that lost connectivity failed their status checks and Auto Scaling groups terminated and replaced them. In US1 more than 8 percent of instances were terminated every 10 minutes between 06:00 and 07:00 UTC, and by 08:00 AWS had replaced about 60 percent of the region's instances, tens of thousands. Stateless services there recovered fast, so the first assessment was that Google Cloud and Azure were in worse shape; it later became clear that AWS was harder to recover, because the data on the terminated instances' local disks was gone.
On an affected node, ip rule listed only "32766: from all lookup main" and "32767: from all lookup default". The rules Cilium installs, such as "100: from all lookup local" and per-pod source routing rules like "111: from 10.132.126.226 lookup 11", were gone. ip route get showed pod traffic to 8.8.8.8 leaving through the host interface ens5 instead of the pod interface ens6, and reply traffic addressed to the host IP being sent back out through ens5 instead of being delivered locally, so the cloud network dropped both.
Root cause ? Breaking change in an upgrade
–
–
A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change.
Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.
same root cause · shared Node operating system image, systemd-networkd
same root cause · shared Node operating system image
same root cause · shared Node operating system image