IL-0168 Scaleway · 2024-02-13kubernetes
Scaleway's managed Kubernetes services Kapsule and Kosmos failed in the Paris region (FR-PAR) on 13 February 2024 from 01:04 until 16:40 CET. A new Kubernetes release had become available to all users on 12 February, and overnight many clusters triggered their auto-upgrade simultaneously. The resulting load made the Kubernetes API slow and congested, and after RAM and CPU spikes it ran out of memory. Scaleway later named the root cause as insufficient resources allocated to the Kubernetes API server.
With the API unavailable, nodes failed to authenticate to their control plane and became NotReady. A new node authentication mechanism, deployed on 24 January 2024, failed and retried continuously, which deepened the congestion, and auto-healing replaced some nodes automatically, which amplified the problem. The same fail-and-retry congestion had caused an incident on 24 January 2024, and the short-term mitigations put in place since then had not prevented it from happening again.
Scaleway raised the API's resources twice, applied API traffic restrictions at 02:30 to smooth incoming traffic, and from 03:20 removed them progressively in 50-100 batches, after which nodes authenticated and became Ready. A script launched at 03:40 replaced the failing nodes created during the congestion, and service was stable by 05:30. At 15:45 Scaleway began rolling back to the former authentication mechanism on a subset of clusters; congestion returned at 16:00 and was handled the same way, and all nodes were Ready by 16:40. Cluster upgrade and configuration update features were disabled to avoid simultaneous node authentication. The rollback reached all clusters at 19:41 on 19 February, the features were re-enabled at 10:30 on 20 February, and the incident was closed at 17:56 that day.
At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.
Kubernetes API response times rose until some requests timed out. The auto-healing feature automatically replaced some of the failing nodes, which made the problem bigger.
The first increase of the API's resources, at 01:46, seemed to help but the API ran out of memory again; a second increase at 02:20 brought improvement. Later in the day, when the rollback of the node authentication mechanism started at 15:45, API congestion returned at 16:00 and nodes again went Not Ready.
Symptom ? Node NotReadyControl plane lagOOMKilled
Root cause ? Capacity exhaustion
The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.
Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem.
The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server