IL-0012 Omio · 2020-02-17 (published)kubernetesgcp
A CPU limit is enforced as a time quota per scheduling period, so a container with many threads can spend its quota in a fraction of the period and then sits paused for the rest. A known kernel bug made the throttling worse than the configured limit should allow, by up to half again. Kubernetes exposes usage, not throttling, so the stalls were invisible until the team read the cgroup statistics inside the pods. They removed CPU limits across their clusters until a patched kernel could be rolled out, relying on tuned runtime defaults in their base images to keep containers from hogging nodes. Error rates and latency dropped at once.
For about six months applications froze now and then or stopped answering health checks with no clear reason. Connections between services broke or timed out, and error rates and latency on important services were higher than they should have been.
Symptom ? CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeouts
Root cause ? Software defect
Trigger Multi-threaded workloads running under CPU limits on nodes whose kernel still had an unpatched scheduler quota bug.
Stalls and failed health checks, then cgroup throttling counters read inside pods, with graphs of error rate and latency.
Requests and limits look alike but are implemented differently: a limit is a time quota that can stall a multi-threaded container well before it uses the configured cores. Removing limits trades away protection against a noisy container, which is only safe with tuned runtimes.
–