IL-0097 2i2c · 2026-03-31kubernetesaws
The berkeley-geojupyter cluster ran an older Kubernetes version, 1.33, than most of 2i2c's clusters. When the upgrade batches were designed, only clusters running 1.32 were taken into account, so this cluster was missed. A day before the incident an upgrade bumped the cluster autoscaler version, and the new autoscaler did not work with Kubernetes 1.33: it triggered no scale-up events, so no new nodes were spawned and servers on the prod hub failed to start.
Because the context of the autoscaler bump was fresh, the team went down the right path from the start. At 15:41 the team decided to upgrade the cluster to Kubernetes 1.34, and then downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34. At 16:48 the team confirmed this fixed it. The report gives the impact time as 15:10 to 17:00 and the duration as 1h 49m 10s.
At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.
Six minutes later an engineer saw no scale-up event for the cluster and started checking the node group labels. New nodes were not being created because the cluster autoscaler was not triggering scale-up events. Another engineer noticed that the cluster was still running Kubernetes 1.33, and the autoscaler deployment logs were checked next because the autoscaler did not seem to behave properly.
At 15:37 the engineer who had merged an upgrade the day before connected the failure to it: that upgrade had also bumped the cluster autoscaler version.
Symptom ? Scale-up failure
Root cause ? Breaking change in an upgrade
An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.
Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler.
2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.
–