IL-0163 Linode · 2023-05-31kubernetes
A change to the API made LKE clusters scale improperly. A gap in Linode's automated testing did not cover this failure case, so the change reached production at about 16:00 UTC on 31 May 2023. Affected customers' node pools were resized wrongly and accrued charges for the extra nodes.
Linode identified the cause at 08:56 UTC on 1 June and deployed a hotfix at 09:40 UTC. The extra worker nodes were left in place so that customer workloads would not be disrupted, and affected customers were contacted by ticket about credit adjustments and further mitigation. The incident was marked resolved at 21:23 UTC on 1 June.
Resizing an LKE cluster produced an incorrect number of nodes, so some customer clusters were resized in unexpected ways and the extra worker nodes were billed to the customer accounts. The clusters themselves kept functioning. Linode's first status update, at 05:07 UTC on 1 June 2023, advised customers to avoid resizing their clusters if possible.
Symptom ? Autoscaler misbehaviour
Root cause ? Faulty deployment
The first status update, at 05:07 UTC on 1 June, said Linode had been made aware of an issue when resizing LKE clusters, about 13 hours after the change went out.
An API change that alters scaling behaviour resized customer clusters and added cost while the clusters themselves kept working; Linode announced the issue about 13 hours after the change went out. Removing the extra nodes would have disrupted workloads, so the cost was handled through credits instead.
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool