IL-0077 Marcin Cuber · 2024-06-05 (published)kubernetesaws
EKS provisions etcd with an 8GB database, the upstream recommended maximum. When etcd passes its size limit it raises a no space alarm and refuses writes, and because compaction is also blocked while the alarm is active, deleting objects cannot free space. In this incident the author found that 8GB on EKS behaved as a soft limit and the cluster also breached a 10GB hard limit, which locked the control plane managed in AWS's account. The objects filling etcd were ephemeralreports created by Kyverno, the policy engine; the Kyverno team later confirmed a bug in 1.12 that let ephemeralreports pile up and released a fix in 1.12.4.
Because the EKS control plane is managed by AWS, the author could not reach etcd and opened a business-critical AWS support case. Support first raised the etcd storage to 12GB, which unlocked writes but left kubectl commands not taking effect, with the API server lagging and etcd struggling under the leaked objects. After hours of investigation an AWS engineer, with the necessary internal approvals, deleted the leaked objects directly from etcd, and the cluster started working again. The author suspects a command like etcdctl delete with a prefix on /registry/reports.kyverno.io/ephemeralreports/trading was used.
The incident was a P1 and took over seven hours. The author had asked AWS support several times, including authorising it in writing, to delete the objects from etcd straight away, and believes the outage could have been about 30 minutes.
A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.
When the etcd database exceeds its size limit, the API server returns errors that appear in the EKS audit logs in CloudWatch as responseObject.code 500 with the message "etcdserver: mvcc: database space exceeded". The author shows a CloudWatch Logs Insights query on the kube-apiserver-audit log stream that filters for "mvcc: database space exceeded" to find when the error first appeared.
The author identified the objects taking up etcd space as Kyverno's ephemeralreports. A script that deletes them in batches of 1000 through the raw API did not help, because etcd was already completely locked.
Root cause ? Software defect
The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.
On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server
same root cause · shared etcd
same root cause · shared Kubernetes API server