IL-0194 Palark · 2023-03-23kubernetes
The DaemonSet was the Vector log collector, which uses pod data to add metadata to log lines. Each of its pods asked kube-apiserver for the pods on its own node with a field selector, a request that returns at most 110 pods. etcd, however, stores objects as plain key-value pairs under /registry/<kind>/<namespace>/<name> and knows nothing about field or label selectors, so kube-apiserver had to read every pod in the cluster from etcd (7130 in the logged request) and filter them itself. With the default resourceVersion behaviour (MostRecent), every such list goes to etcd. When all the collector pods started together after a kube-apiserver restart, these reads drove etcd memory up and brought the master nodes down.
Palark first tried resourceVersion=0, which lets kube-apiserver answer from its cache: a test list of all pods took 117 ms instead of 337 ms. The collector is written in Rust, and the Kubernetes Rust library had no resource version setting, so this was not available right away; Palark later submitted a pull request to the library. To end the incident, Palark used API Priority and Fairness to queue the requests instead. A PriorityLevelConfiguration named limit-list-custom (type Limited, assuredConcurrencyShares 5, queuing with 16 queues, handSize 4, queueLengthLimit 50) and a FlowSchema that sends list and get requests on pods from the collector's service account to that level, distinguished by user, were applied. After that, etcd memory was mostly normal with some spikes left, and the incident's criticality was lowered.
Palark found a similar problem with the Cilium CNI and handled it the same way.
One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.
etcd logged slow reads over the whole pod key range, for example a warning "apply request took too long" with "took": "130.768157ms", "expected-duration": "100ms", a read-only range from "/registry/pods/" to "/registry/pods0" and "range_response_count:7130 size:13313287".
The audit logs pointed to one DaemonSet. Whenever kube-apiserver restarted, each of its pods sent new list requests to fill its caches, which is the default behaviour of client-go informers: 60 list requests per pod at startup, with about 80 nodes in the cluster. The requests had the form /api/v1/pods?fieldSelector=spec.nodeName=$NODE_NAME. Redeploying the pods one after another left no mark on the memory graphs; only restarting them all together did.
Root cause ? Capacity exhaustion
Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.
A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.
same root cause · shared etcd, Telemetry agent, Kubernetes API server
same root cause · shared etcd, Kubernetes API server
same root cause · shared etcd, Kubernetes API server
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server