IL-0195 Palark · 2023-06-07 (published)kubernetes
A customer cluster run by Palark, with more than 200 nodes, lost its Kubernetes API when the Cilium agent pods sent LIST requests to the API en masse. Simultaneous LIST requests from so many nodes drove kube-apiserver memory up until one API server used all the memory on its control plane node and crashed; its requests then moved to the next API server, which failed the same way.
To keep production running, the team first added CPU and memory to the control plane nodes. The first increase was not enough, and the API stabilised only after the next one; in total the nodes went from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB.
The lasting fix used API Priority and Fairness (APF), the API server feature that sorts incoming requests into priority levels, each with its own share of concurrency and its own queues. A FlowSchema named cilium-pods matched LIST requests to the cilium.io API group from the group system:serviceaccounts:d8-cni-cilium, split them into flows by user, and sent them to a PriorityLevelConfiguration of type Limited with assuredConcurrencyShares 5, 16 queues, handSize 4 and queueLengthLimit 50. The team accepted slightly slower LIST responses for Cilium because they would not affect Cilium's work. With the new objects in place, restarting cilium-agent no longer changed memory use significantly, and the control plane nodes went back to their original size.
A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.
Memory graphs from the time of the failure showed memory consumption as high as 50 GB. The control plane nodes originally had 8 CPUs and 16 GB of RAM each. In a reply to a reader, the team explains how the source was found: a Grafana dashboard of API request counts showed a large increase, the Prometheus query apiserver_request_total{verb=~"LIST", instance=~".*:.*", resource=~".*"} grouped by instance and resource showed that many LIST requests came from Cilium network policies, and the API audit log showed the individual requests.
In a test window agreed with the customer, the team restarted the Cilium agents and watched the same sequence repeat: the load on one API server grew, it consumed all the memory on its node and crashed, the requests moved to another API server, and the same thing happened there.
Symptom ? Control plane unresponsiveOOMKilled
Root cause ? Capacity exhaustion
A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.
In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once.
With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server