IL-0157 Linode · 2025-03-05kubernetes
Akamai traced the failure to the latest software release deployment. A software defect in that release kept new nodes from joining LKE clusters that had IP ACL enabled, so the cluster autoscaler on those clusters could not add nodes. Restarting the gateway pods helped only for a short time. Akamai rolled back the release, the rollback was complete on all affected clusters at about 15:00 UTC, and the incident was marked resolved at 15:12 UTC. The postmortem, posted on 2025-03-12, says Akamai was still investigating the root cause.
From about 01:00 UTC on 2025-03-05, Linode Kubernetes Engine (LKE) clusters with IP ACL enabled failed to autoscale: new nodes could not join the cluster, and some customers had service disruptions. About 20 to 25 clusters in several regions around the world were affected; the status page lists the US-IAD (Washington), US-ORD (Chicago), US-SEA (Seattle) and IN-MAA (Chennai) LKE components.
Restarting the gateway pods was tried as a temporary mitigation and gave only short-term relief.
Symptom ? Scale-up failureService outage
Root cause ? Software defect
Customers began having problems at about 01:00 UTC, and the first status post, at 06:49 UTC, said the team was investigating an LKE autoscaling problem when IP ACL is turned on.
The defect hit only clusters with IP ACL enabled, and it showed when the autoscaler tried to add nodes. Restarting the gateway pods did not hold; rolling back the release ended the incident.
same root cause · shared Cluster autoscaler