IL-0181 Kevin · 2026-02-19 (published)kubernetes

A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

The product service pods ran with a CPU request of 500m and a CPU limit of 1 core. When the sudden traffic hit the product lookup API, the containers reached that limit and the Linux CFS (Completely Fair Scheduler) held back their CPU use, so requests were not processed in time even though the pods looked healthy. The author had believed the resource settings could absorb the event traffic the team expected, and a single external post pushed traffic beyond that.

The author first raised minReplicas of the HorizontalPodAutoscaler for my-service-prd from 5 to 10 (maxReplicas 20, target average CPU utilization 70%). With the same traffic spread over 10 pods, per-pod CPU use dropped and response times returned to normal. The author then raised the container CPU limit from 1 core to 2 cores, noting that the base setting had been 3 cores, and plans to keep adjusting the limit through monitoring so that pods do not take too much node capacity.

Severity: –
Scope: product lookup page and product API of an e-commerce platform
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

The Kubernetes dashboard showed every pod Running with no restarts. The CPU usage of the affected service's namespace was close to 100%. The pods were not restarted the way an OOMKill would restart them and the logs showed no clear error; looking at the pod list alone, the author did not see the cause. Only the CPU usage metrics showed that the containers were being throttled.

Symptom ? CPU throttling High latency

Root cause

Root cause ? Capacity exhaustion

Component ?Container CPU limit Horizontal pod autoscaler

Contributing factor ?Monitoring blind spot

Trigger: A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

Detection

Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

Remediation

raised the HPA minReplicas from 5 to 10
raised the container CPU limit from 1 core to 2 cores

Action items

Keep adjusting the CPU limit to a suitable level through monitoring planned

Lessons

CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do.

Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

Sources

https://velog.io/@yardyard/Kubernetes-CPU-Throttling-%EC%9E%A5%EC%95%A0-%EB%8C%80%EC%9D%91%EA%B8%B0 · blog

Related cases

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
same root cause · shared Horizontal pod autoscaler
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024
same root cause · shared Horizontal pod autoscaler
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023
same root cause · shared Container CPU limit