IL-0149 Microsoft Azure · 2019-09-28kubernetesazure

Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy

On 3 October 2019 AKS engineering opened a known-issue report: clusters using VMSS that were created before 07:20 UTC on 28 September 2019 failed to upgrade, because the node image the scale set referenced was reported as not available. The preliminary root cause was a recent service update that left instances of a backend service unhealthy, so requests could not complete. Customers were told not to upgrade these clusters until the issue was mitigated.

Customers could fix a cluster themselves with two az vmss update commands, setting VirtualMachineProfile.StorageProfile.ImageReference.version to "2019.09.25" and VirtualMachineProfile.StorageProfile.ImageReference.Sku to "aks-ubuntu-1604-201909" on the scale set in the MC resource group, and then running the upgrade again. Those who could not were asked to file a support request. The AKS and VMSS teams mitigated affected clusters by hand and rolled hot fixes out region by region; 14 regions were patched by 4 October, and the issue was closed on 15 October 2019 when all regions were patched.

The issue title says clusters created before 20 September, while its body gives 07:20 UTC on 28 September 2019.

Severity: –
Scope: Upgrades of all AKS clusters on Virtual Machine Scale Sets created before 07:20 UTC on 28 September 2019
Users affected: –
Region: All regions
Time to detect: –
Time to resolve: –

Symptoms

From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

All VMSS-based clusters created before that time were affected, in all regions, and could not be upgraded without manual intervention. A customer asked in the issue how to tell whether a cluster was affected without trying an operation and having it fail; the issue gives no answer.

Symptom ? Upgrade stall

Root cause

Root cause ? Faulty deployment

Component ?Node operating system image Node pool

Contributing factor ?–

Trigger: A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

Detection

Customers received failure messages when they tried to upgrade their clusters.

Remediation

published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
mitigated affected clusters by hand (AKS and VMSS teams)
rolled out hot fixes to all regions, completed by 15 October 2019

Action items

–

Lessons

–

Sources

https://github.com/Azure/AKS/issues/1250 · github

Related cases

IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026
same root cause · shared Node pool
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025
same root cause · shared Node pool
IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accountsLinode · 2023
same root cause · shared Node pool
IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silentlyLinode · 2022
same root cause · shared Node pool