IL-0144 Microsoft Azure · 2022-08-30kubernetesazurenetworking
At 06:00 UTC on 30 August 2022 Canonical published an Ubuntu security update. Azure VMs on Ubuntu 18.04 with unattended-upgrades enabled, which by default downloads and applies security updates once a day, installed it, including systemd 237-3ubuntu10.54. A race-condition bug in that systemd version (Launchpad bug 1988119), set off by this update combined with an earlier one, made the VMs lose their DNS configuration. The bug only affects systems whose network configuration identifies the network interface card by driver name, which is why it hit Azure and not other major cloud providers.
Because of their criticality, security updates like this one do not go through Azure's Safe Deployment Practices process; Azure says it is reviewing that process. AKS nodes on Ubuntu 18.04 lost DNS, kubelet could not resolve registry hostnames, and new pods could not pull their images. Downstream services built on these VMs, including Azure Monitor, Application Insights, Log Analytics and Microsoft Sentinel, were affected by the same DNS failures.
Mitigation was done partly together with Canonical. Azure pointed customers to Canonical's description of the bug and a fix, and recommended rebooting affected VMs or, where that was not possible, running a template script to repair DNS after testing it. For AKS nodes, the AKS team built automatic detection and remediation and rolled it out in all regions, which resolved the vast majority of customers; the rest needed manual mitigation through support. Impact lasted until 16:00 UTC on 31 August 2022.
From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.
On AKS, kubelet could not resolve the DNS names of container registries, so pod creation failed with errors such as ImagePullBackoff. Other Azure services that run on Ubuntu 18.04 VMs failed the same way: customers of Azure Monitor, Application Insights, Log Analytics and Microsoft Sentinel may have been unable to reach their log data and may have seen Log Search alerts and Microsoft Sentinel alerts missed or delayed.
The problem was limited to Ubuntu 18.04 but appeared in every Azure region, public and sovereign clouds alike. Incoming customer support cases confirmed that only those Ubuntu versions were affected.
Symptom ? DNS resolution failuresImage pull failuresMonitoring outage
Root cause ? Breaking change in an upgrade
Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.
Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.
same root cause · shared Client-side DNS resolver and cache
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node operating system image