IL-0148 Microsoft · 2024-07-24 (published)kubernetesazure

A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters

The AKS team found a regression in kube-scheduler that affected Kubernetes v1.27.14, v1.28.10 and v1.29.5. With the bug, a single pod whose node affinity points at a nonexistent node made the scheduler panic, which stopped scheduling for every pod in the cluster. The bug is upstream, tracked as kubernetes/kubernetes#124930.

The AKS team announced the bug in a pinned GitHub issue on July 24, 2024. AKS engineers fixed the affected clusters and rolled out a fixed kube-scheduler to all regions, applied automatically without customer action. The issue was closed on August 29, 2024, with a note that the upstream fix had shipped in the July AKS releases.

Severity: –
Scope: AKS clusters on Kubernetes v1.27.14, v1.28.10 or v1.29.5 that contained a pod with a node affinity naming a nonexistent node: no pods were scheduled
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

On AKS clusters running Kubernetes v1.27.14, v1.28.10 or v1.29.5, kube-scheduler panicked whenever the cluster contained a pod with a bad node affinity, and from then on no pod in the cluster was scheduled.

The AKS team's example of a pod that triggers the bug sets a required node affinity (requiredDuringSchedulingIgnoredDuringExecution) with a nodeSelectorTerms entry using matchFields on the key metadata.name, operator In, and the value "invalid-node", a node that does not exist.

Symptom ? Pods Pending