Deferred upgrades

Cases

1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created2i2c · 2026

   Root cause Breaking change in an upgrade

2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1

   Root cause Breaking change in an upgrade

3. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025

   Root cause Software defect

4. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in productionSunwoo Park · 2023

   Root cause Breaking change in an upgrade

5. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutesDFDS · 2022

   Root cause Certificate or credential expiry

6. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

   Root cause Operator error