IL-0161 Linode · 2024-04-22kubernetes
From 22 April 2024, new LKE clusters in the Toronto data center failed to provision. Clusters were created without node pools, or with the wrong number of nodes, and the Cloud Manager timed out when customers created them. The cause was an earlier API update that did not work as intended and led to timeouts during provisioning.
At 17:32 UTC on 22 April the team narrowed the problem down to a specific section of infrastructure in Toronto. At 15:03 UTC on 23 April that infrastructure was taken out of use for new LKE clusters, to stop further customer impact while the investigation continued. A fix for the root cause went in at about 16:45 UTC on 25 April, and by 18:05 UTC cluster creation was confirmed to be fully restored. The disabled infrastructure was then put back into use, and after a period of monitoring the incident was closed at 17:09 UTC on 26 April.
At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.
A customer who created a new LKE cluster saw a timeout in the Cloud Manager. After refreshing the Cloud Manager, the new cluster was listed but had no node pools. Other newly provisioned clusters had an incorrect number of nodes in their pool, or no nodes at all.
The errors happened mostly in one distinct section of the infrastructure, a specific backend resource in the Toronto data center. Much more rarely, the same errors appeared in other data centers.
Symptom ? Cluster or node provisioning stallIntermittent timeouts
Root cause ? Software defect
Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.
–
The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool