IL-0028 Temporal · 2023-06-13 (published)kubernetesobservability
The post looks at request latency of a self-hosted Temporal Server on Kubernetes, using the History service as the example. Kubernetes turns a container's CPU limit into a quota of CPU time per 1/10th of a second; once the container uses the quota up, its execution waits for the next period. That delay adds latency, most of all to CPU-heavy work such as taking locks. On Temporal's own Kubernetes cluster the nodes had 8 cores and the history pods were limited to 2.
Temporal Server is written in Go, and GOMAXPROCS, the Go runtime setting that controls how many processes Go may use for concurrent work, defaults to the number of cores on the machine. Each History container therefore ran work across up to 8 cores while only 2 cores' worth of time was allowed per period. Garbage collection and other housekeeping ran on cores the container could not really use, and the container's processes starved each other of the allowed CPU time. Setting the GOMAXPROCS environment variable to 2 made the throttling disappear, made latency steadier and slightly lower, and reduced CPU usage across all Temporal services. From release 1.21.0 Temporal sets GOMAXPROCS from the Kubernetes CPU limit when a limit exists and the variable is unset; it does not look at CPU requests, so without limits GOMAXPROCS should be set by hand to the CPU request or slightly above.
The author also recommends dropping CPU limits and sizing CPU requests carefully while watching node CPU, or, where limits are required for reasons such as customer isolation, putting Temporal on dedicated nodes with taints and tolerations. For the latency spikes during rollouts: with more than 10 history pods, roll one pod at a time with low maxSurge and maxUnavailable and give each pod time to rebalance with minReadySeconds or a startupProbe; with fewer pods, roll quickly. Temporal had good results with 50 percent maxSurge and maxUnavailable at 4 pods or fewer. The Matching service, which balances task queue partitions, behaves the same way.
Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.
The CPU usage graph hid the problem because of a gap in time scales. Kubernetes enforces a container's CPU limit as a budget of CPU time in every 1/10th-second period, while Prometheus in the kube-prometheus stack scrapes aggregated CPU metrics every 15 seconds or more. A container can run out of its budget in many short periods and still average far below 100 percent over 15 seconds. The throttling only showed up in a query that divides container_cpu_cfs_throttled_periods_total by container_cpu_cfs_periods_total for the temporal-history workload in the temporal namespace.
Separately, latency could spike when the History deployment was rolled out, because each pod added or removed makes the service move history shards between pods, and a shard cannot be written while it is being reassigned. Metrics emitted by the History service itself are unreliable during a rollout, since Prometheus may not scrape new pods for a few intervals after they start.
Symptom ? CPU throttlingHigh latency
Root cause ? Capacity exhaustion
A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.
A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.
same root cause · shared Container CPU limit
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Prometheus
same root cause · shared Prometheus