IL-0175 Hyperconnect · 2022-07-19 (published)kubernetes
Hyperconnect runs most of its microservices on Kubernetes. The service's deployment ran with -XX:+UseContainerSupport, heap set to 60% of RAM and a 1Gi memory limit, and deliberately without a CPU limit, because a regular JVM uses a lot of CPU at start-up and while it compiles classes for the first requests, and a CPU limit makes start-up slower and early latency spiky.
The JVM's container support (Corretto 11 in this case) computes the processor count from the cgroup's CPU quota, period and shares. With no CPU limit, the quota was -1, so only cpu.shares counted, which is 1024 with one requested core and 2048 with two. The HotSpot function CgroupV1Subsystem::cpu_shares() treats a value of exactly 1024 as no shares set and returns -1, so with one requested core the JVM fell back to the CPU count of the node, 36. Spring WebFlux then started 36 reactor-http-epoll threads instead of 4, and the extra thread memory pushed the pod over the alert threshold.
Raising the request again, setting a CPU limit, or requesting 1001m or 999m each had its own drawbacks, so the team fixed the processor count explicitly with -XX:ActiveProcessorCount=1 in JAVA_OPTIONS and wrote an internal notice. The issue is tracked in JDK-8281571 and JDK-8281181; the fix, merged in OpenJDK 18 and planned for JVM 19 with a UseContainerCpuShares option for compatibility, stops CPU requests from affecting availableProcessors() at all, so a CPU limit or ActiveProcessorCount has to be set explicitly.
During a DevOps review of resource usage across microservices, the team lowered resources.requests.cpu from 2 to 1 for notification-preferences, a Spring WebFlux service not yet used in production. A few minutes later an alarm fired because memory usage was above 90%. Neither the memory allocation nor the JVM's GC arguments had been changed.
GC metrics collected through JMX showed no change. The JVM metrics did show that the thread count had more than doubled; about 20 extra threads seemed enough to make a difference of around 100 MB in a pod with 1 GB of memory. VisualVM comparisons before and after the change showed that the extra threads were named reactor-http-epoll. Reactor Netty sizes this pool as the larger of availableProcessors() and 4, so it was 4 with two CPUs requested but 36 with one.
In jshell inside the pod, Runtime.getRuntime().availableProcessors() returned 2 with requests.cpu set to 2, but 36 with requests.cpu set to 1. 36 is the CPU count of the whole node. With requests.cpu at 1001m, /sys/fs/cgroup/cpu/cpu.shares read 1025 and availableProcessors() returned 2; at 999m, cpu.shares read 1022 and it returned 1.
Symptom ? Memory pressure
Root cause ? Software defect
A memory usage alarm above 90% fired a few minutes after the CPU request was lowered.
On cgroup v1, a JVM in a pod with a CPU request of exactly one core and no CPU limit sees every CPU of the node, because cpu.shares 1024 is read as unset. Thread pools sized from availableProcessors(), such as Reactor Netty's I/O workers, grow accordingly and use more memory.
When lowering requests for JVM services without a CPU limit, set -XX:ActiveProcessorCount or a CPU limit explicitly. Later JDKs ignore CPU requests for this calculation altogether.
same root cause · shared Container CPU limit
same root cause · shared Container CPU limit