IL-0146 Microsoft Azure · 2020-12-04kubernetesazurenetworking

New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI

From about 00:00 UTC on 4 December 2020 to 00:10 UTC on 15 December 2020, newly created AKS clusters with Azure CNI in Central India could have tunnel connectivity problems. VMs in the region began to receive MAC addresses with a 06 prefix instead of the usual 00, and that revealed a bug in Azure CNI that had been dormant for a long time. The newest Azure CNI, which no longer uses a bridge, was not affected.

The AKS team published a way to check each node with az vmss run-command (or az vm run-command for availability sets) for the MAC address and the CNI mode. The fix was to get the newer CNI: on scale sets a node image upgrade, or any Kubernetes upgrade such as a patch version; on availability sets, scaling out and deleting the affected VMs; for a brand-new cluster, recreating it. A hotfix to make transparent mode the default in all regions was in progress; existing clusters still had to be upgraded.

Severity: –
Scope: Newly created AKS clusters using Azure CNI in bridge mode; clusters using kubenet were not affected
Users affected: –
Region: Central India
Time to detect: –
Time to resolve: 11d 10m

Symptoms

In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.

On an affected node, ip addr show eth0 showed a MAC address beginning link/ether 60:45:.. and /etc/cni/net.d/10-azure.conflist showed "mode":"bridge". A node with link/ether 00:0d:.. in bridge mode, or 60:45:.. in "mode":"transparent", was not affected.

Symptom ? Intermittent timeouts

Root cause

Root cause ? Software defect

Component ?Azure CNI Node operating system image

Contributing factor ?–

Trigger: VMs in Central India started being assigned MAC addresses in a much higher range than before (06 prefix rather than the usual 00).

Detection

–

Remediation

told owners of affected scale-set clusters to run a node image upgrade, or any Kubernetes upgrade such as a patch version, to get the Azure CNI that does not use a bridge
told owners of availability-set clusters to scale out and delete the VMs with a high-range MAC address in bridge mode
advised recreating affected clusters that were brand new

Action items

Roll out a hotfix that makes transparent mode the default in all regions planned

Lessons

A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.

Sources

https://github.com/Azure/AKS/issues/2023 · github

References

https://docs.microsoft.com/en-us/azure/aks/node-image-upgrade

Related cases

IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025
same root cause · shared Node operating system image
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025
same root cause · shared Node operating system image
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022
same root cause · shared Node operating system image
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020
same root cause · shared Node operating system image
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019
same root cause · shared Node operating system image