IL-0020 UK Ministry of Justice · 2025-03-18kubernetesaws
With the old nodes cordoned, the descheduler tried to move every pod off them at once while some of the new nodes were stuck, so evicted pods had nowhere to go. Removing the cordon and deleting the stuck nodes brought pods back within an hour and a half. The team then added a twenty-node group by hand for headroom and drained the old nodes in a controlled way; the cluster was fully upgraded by late afternoon and the incident closed two days later after cleanup. The descheduler's behaviour toward cordoned nodes is still to be understood.
Users reported pods being terminated and not coming back in production and non-production alike, and infrastructure alerts fired at the same time. Several production services were down.
Symptom ? Pod evictionPods PendingService outageUpgrade stall
Root cause ? Faulty deployment
Trigger Rolling the live cluster from one Kubernetes minor version to the next, which cordoned the old nodes.
Automated infrastructure alerts and user reports.
The team needs a documented way to add emergency capacity and alerts on node-count imbalance and pending-pod spikes so the failure is caught before users report it.
–