IL-0029 Buffer · 2020-08-31 (published)kubernetesaws
Kubernetes enforces a container's CPU limit with the kernel's CFS quota: in each period (100ms by default) a container may use only its share of CPU time and is throttled after that. Buffer had set CPU limits on its services because that was the common recommendation, and because without limits a busy container had earlier starved the kubelet and turned nodes NotReady. The team then saw throttling on containers whose usage was far below the limit, and traced it to a Linux kernel bug that throttled containers with CPU limits unnecessarily. The fix for that bug is in kernel 4.19 and later, but the kops version Buffer ran still used an older host image, so upgrading the kernel was not yet possible.
Instead, after long discussion, the team removed CPU limits from every service that sat directly or indirectly on the users' critical path, and did it with safeguards. Those services were moved to dedicated tainted nodes, with tolerations, so that an unbounded service could only disturb other unbounded ones. Each service's CPU request was set from a few months of Datadog usage data: the peak usage plus a margin of more than 20%, with memory requests set the same way, so the scheduler reserved enough room on the node. The cost was lower container density and idle capacity at quiet hours.
After a few weeks, latency improved across all the modified services; the main landing page buffer.com became up to 22 times faster. The author adds that upgrading to a fixed kernel should be preferred over removing limits, and that he could not yet confirm the kernel fix removes throttling completely.
Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.
One of the main APIs is the example in the write-up: its CPU limit was 800m, its peak usage was at most 200m, and the throttling graph still showed throttling. On paper the service had four times the CPU it needed, yet the kernel was holding it back.
The opposite setting had already hurt the team before. When containers ran without CPU limits and used up a node's CPU, the kubelet on that node stopped responding, the node went NotReady, and the pods that were moved elsewhere carried the same problem onto the next nodes. That history is why removing the limits was a long discussion and not a quick fix.
Symptom ? CPU throttlingHigh latencyNode NotReady
Root cause ? Software defect
While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.
A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.
same root cause · shared Node operating system image, Linux kernel
same root cause · shared Container CPU limit, Linux kernel
same root cause · shared Node operating system image, Linux kernel
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Linux kernel
same root cause · shared Container CPU limit
same root cause · shared Node operating system image