IL-0165 Linode · 2023-05-25kubernetes

Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints

Starting at 14:00 UTC, backend services of the Linode Kubernetes Engine in Frankfurt failed because of memory constraints, and a portion of customers lost access to their kube-apiservers through kubectl. Administrators identified the cause at 14:27 and raised the memory limits at 14:29.

By 14:34 Linode confirmed a full recovery and that customers could reach their kube-apiservers again. The postmortem says Linode is taking further steps to increase resilience, redundancy and monitoring for the affected LKE components.

Severity: –
Scope: access to kube-apiservers through kubectl for a portion of LKE users in the Frankfurt data center
Users affected: a portion of LKE users in the Frankfurt data center
Region: EU-Central (Frankfurt)
Time to detect: 11m
Time to resolve: 34m

Symptoms

At 14:11 UTC on May 25, 2023, Linode learned that part of its LKE users in the Frankfurt data center could not reach their kube-apiservers through kubectl. The status page opened at 14:27 UTC with "We are investigating reports of connectivity issues with the LKE service within the Frankfurt Datacenter".

Symptom ? Control plane unresponsive

Root cause

Root cause ? Capacity exhaustion

Component ?Kubernetes API server Container memory limit

Contributing factor ?–

Trigger: –

Detection

Linode was made aware of the problem at 14:11 UTC; the status page refers to reports of connectivity issues.

Remediation

increased memory limits

Action items

Increase resilience, redundancy and monitoring for the LKE components affected by the incident planned

Lessons

–

Sources

https://status.linode.com/incidents/hxxfnyhfprlj · official-status

Related cases

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
same root cause · shared Container memory limit
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025
same root cause · shared Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025
same root cause · shared Container memory limit
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024
same root cause · shared Kubernetes API server
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024
same root cause · shared Container memory limit
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024
same root cause · shared Container memory limit
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024
same root cause · shared Kubernetes API server
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady