IL-0147 Microsoft · 2025-08-26kubernetesazurenetworking
AKS node images (VHDs) pre-cache the Kubernetes node binaries for the two most recent patch versions of each minor release. When a node boots on a version that is not cached, the Custom Script Extension (CSE) bootstrap downloads the binaries from acs-mirror.azureedge.net. A recent refactoring in the bootstrap stopped the HTTP proxy configuration from being used for that download, so in clusters where nodes can reach the internet only through a proxy, the download tried direct egress, timed out, and the CSE exited with code 31 once it passed its 780-second limit. Node provisioning failed.
The affected node image versions were 20250806, 20250811 and 20250820. Clusters on a version whose binaries were cached in the image, such as v1.31.9 and v1.31.10 in 20250806 and 20250811, were not affected, which is why the bug showed only on AKS 1.31.0 to 1.31.8 and other uncached versions. The issue was opened on 2025-08-27; the AKS team confirmed the cause on 2025-09-03, and VHD 20250811 with a hotfix was globally available in public regions on 2025-09-04.
Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.
The VMSS activity log showed "VM has reported a failure when processing extension 'vmssCSE'" with "command terminated with exit status=31". The CSE output showed curl repeatedly trying to download https://acs-mirror.azureedge.net/kubernetes/v1.31.8/binaries/kubernetes-node-linux-amd64.tar.gz by connecting straight to the IP addresses of acs-mirror.azureedge.net on port 443, each attempt ending in a 60-second timeout, until "Error: CSE has been running for 806 seconds, exceeding the limit of 780 seconds." and "CSE timeout approaching, exiting early."
The reporter found that 202507.* images on the same AKS versions worked, and that 202508.* images worked on AKS 1.31.9 and 1.31.10. Another user reported that a cluster on AKS 1.32.4 with node image 202507.21.0 worked, but after a reconcile AKS tried 202508.06.0, provisioning failed and the cluster went into the Failed state.
Symptom ? Scale-up failureUpgrade stall
Root cause ? Software defect
The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.
A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path.
For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.
same root cause · shared Node operating system image
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node pool
same root cause · shared Node operating system image