IL-0145 Microsoft Azure · 2023-08-30kubernetesazure
At 08:41 UTC on 30 August 2023 a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that stopped the chillers in one of the three Australia East availability zones; 13 of them failed to restart automatically. The onsite team restarted chillers by hand on the rooftop, moving from one datacenter to the next, but by the time it reached the last five, the water in their chilled water loop was too hot and a self-protection mechanism blocked the restart. Those five cooled two adjacent data halls that need at least four running chillers. From 10:30 UTC equipment began shutting down automatically as temperatures rose, and at 11:34 UTC the team shut down the remaining networking, compute and storage in the two halls remotely. The chillers came back at 12:12 UTC, temperatures were normal by 13:30 UTC, and power was restored to all hardware by 15:10 UTC.
AKS lost compute for its control plane in Australia East and lost access to the Azure SQL Database it uses as the operation queue for create, read, update and delete activities. The AKS control plane underlay is spread across availability zones, but the SQL Database in Australia East had not been configured with zone resiliency, although it was scheduled to be converted, so it was unavailable for the whole incident. AKS in Australia Southeast depended on the same database, which spread the failure of AKS operations to that region. As SQL Database recovered, the AKS service was restored without any further mitigation.
The rest of the region recovered more slowly in places: most services were back by 22:40 UTC on 30 August, but full mitigation took until 20:00 UTC on 3 September, mostly because of dependencies on recovering storage, SQL Database and Cosmos DB.
From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.
For AKS, operations that go through the AKS resource provider failed: creating, reading, updating and deleting clusters, including scaling. This happened in Australia East and also in Australia Southeast. Workloads already running on AKS clusters in either region were not expected to be affected as long as they did not need those operations.
Symptom ? Control plane unresponsiveHigh error rateService outage
Root cause ? Cloud provider outage
Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.
A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well.
Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database