IL-0014 loveholidays · 2019-03-19kubernetesgcp
Several separate problems piled up during one upgrade. Drains were slow because of emptyDir volumes, long termination grace periods and no disruption budgets. A node-pool operation hung and had to be cancelled, leftover instances had to be deleted by hand, a known ingress bug in the target version broke an internal ingress, and the next day a horizontal autoscaler misbehaved with two active replica sets. The provider later withdrew that patch version for elevated control-plane errors. The team added alerts for autoscalers near their maximum and for deployments with two replica sets.
Node drains took far longer than planned, one node-pool upgrade got stuck and some instances stayed around without terminating. An internal ingress stopped working, and the next day an autoscaler kept scaling a deployment to its maximum.
Symptom ? Upgrade stallService outageAutoscaler misbehaviour
Root cause ? Cloud provider outage
Trigger A manual GKE master and node-pool upgrade from 1.10 to 1.12, two minor versions in one step.
The ingress failure was seen in logs; the autoscaler problem was noticed the next day.
Read the release notes before upgrading, use auto-upgrade where it is safe, and keep environments identical down to resource names.
IL-0020 — linked by the editor