IL-0169 Scaleway · 2024-01-24kubernetes
Simultaneous deployments in FR-PAR drove the Scaleway Kubernetes API, the service behind Kapsule and Kosmos, out of memory. This API was also handling node authentication at the time, so when it failed the kubelets on customer nodes could not renew their leases, the nodes went NotReady, and they retried indefinitely. Requests from these malfunctioning components flooded the API Gateway, which slowed down and timed out on some requests but did not go fully down; changes made after an earlier incident helped it absorb the spike.
On-call engineers took the incident at 00:00:10 and by 00:30 had identified the out-of-memory condition as the cause. Between 00:35 and 00:40 they raised the memory limit and garbage collection thresholds of the Kubernetes managed services and launched more replicas of key services. API Gateway requests began falling fast at 00:43, nodes came back from 00:54 while a heavy queue of instance actions was watched, and from 01:03 all FR-PAR clusters began to stabilise after further steps such as purging the registry cache. Cockpit metrics came back at 01:13.
At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.
Nodes of customer clusters could not authenticate with their control plane, so their kubelets could not update their leases, and some nodes became NotReady and kept retrying without end. Response times of the API Gateway rose until some requests reached timeouts. Where it applied, the cluster auto-healing process started and replaced nodes automatically.
Customers of the managed Kubernetes services Kapsule and Kosmos were affected, in both mutualized and dedicated environments, along with some other products such as Cockpit, whose metrics were missing until the end of the incident.
Symptom ? Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage
Root cause ? Capacity exhaustion
Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.
Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.
same root cause · shared Container memory limit, kubelet
same root cause · shared kubelet
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared kubelet
same root cause · shared Container memory limit
same root cause · shared kubelet
same root cause · shared Container memory limit