IL-0204 Render · 2024-12-17 (published)kubernetesnetworking
Render runs self-managed Kubernetes clusters. In January 2022 an etcd memory spike overloaded the control plane of a cluster in Frankfurt, so deploys and cron jobs failed. The cluster's CoreDNS servers ran on the control plane nodes, and the team rebooted those nodes during the incident. CoreDNS does not need the Kubernetes API server to answer queries, since its kubernetes plugin keeps an in-memory cache, but it does need the API server to initialise. With the API server down, CoreDNS could not start again, and DNS failed for running services too.
The author notes that even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart, because the data plane was not isolated from the control plane. In response to the incident Render moved CoreDNS from control plane nodes to data plane nodes, and moved etcd to dedicated nodes because etcd performs poorly under resource contention, especially when disk write latency rises.
In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.
The incident became far more severe when DNS resolution started failing. From then on, requests to services that were already running failed as well.
Symptom ? Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
Root cause ? Capacity exhaustion
–
–
CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running.
The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.
same root cause · shared etcd, Kubernetes API server
same root cause · shared etcd, Kubernetes API server
same root cause · shared etcd, Kubernetes API server
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared CoreDNS
same root cause · shared Kubernetes API server