IL-0101 Wealthy · 2025-09-14kubernetesgcp
The Wealthy production GKE cluster ran a Kubernetes version close to its deprecation deadline, with node pool auto-upgrade disabled. At the deadline Google upgraded the control plane from v1.32.x to v1.33.2 anyway. Existing nodes kept working, but from then on no new node could initialise properly: NodeLocalDNS crash-looped, the netd network plugin and the CSI driver did not start, and system ConfigMaps did not mount. Because critical background workers ran on spot instances that are preempted often, each preempted node was lost for good, and since no node could be added the cluster could not autoscale at all.
Restarts, rolling back node versions, toggling the NodeLocalDNS addon and recreating the node pool all failed, and the team concluded the cluster could not be repaired. Bringing workloads up on a new cluster also needed component upgrades for Kubernetes 1.33: Istio had to move to 1.27.1 and the cert-manager configuration had to be updated. There was no standby cluster on GCP to fail over to; the dev cluster runs on another cloud provider. The incident lasted 15 hours and was rated Critical.
Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.
On the new nodes the system pods failed: NodeLocalDNS pods were stuck in crash loops, the netd network plugin could not initialise so pods had no networking, the CSI driver could not register so persistent volumes would not mount, and critical system ConfigMaps failed to mount. There were also API server connection errors. The CSI node-driver-registrar logged "Still connecting to unix:///csi/csi.sock" every 10 seconds and then "error connecting to CSI driver: context deadline exceeded".
Restarting failed pods and nodes did nothing, and rolling node versions back still left new nodes unable to initialise. Disabling the NodeLocalDNS addon worked but slowed DNS resolution, and re-enabling it failed with "INVALID_ARGUMENT: Node pool \"t2d-64gb-standard\" requires recreation". Recreating the node pool failed with the same initialisation problems.
Symptom ? Scale-up failureContainer crash loopService outageDNS resolution failures
Root cause ? Breaking change in an upgrade
–
GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures.
Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.
same root cause · shared GKE control plane
same root cause · shared Client-side DNS resolver and cache