IL-0026 Blue Matador · 2019-06-15kubernetesawsobservability
Two production nodes at Blue Matador, on two consecutive evenings in June 2019, reported SystemOOM events. In both cases the memory had been consumed by the fluentd-sumologic pods, the Ruby-based log shipper that forwards logs to Sumo Logic. Their manifest set no resource requests or limits, so nothing stopped them from growing until the whole node was short of memory. Kubernetes resolved each event by killing and restarting those pods, and no application pod was affected.
The first event was dismissed because the check for restarted pods looked only at the application namespace. After the second event the engineer restarted all sumologic pods so they started from low memory, then on Monday added the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 environment variable suggested in a GitHub issue and set memory requests of 128Mi and limits of 1024Mi (CPU 100m and 250m) on the pods. Node memory stayed stable in the following days.
Each SystemOOM was preceded by heavy disk activity and a falling EBS Burst Balance on the node's root volume. Swap was not enabled, and the cause of that disk I/O was never found because the logs from before the events were no longer available.
On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.
The engineer ran kubectl get pods and saw no application pod had died, and kubectl top nodes showed the node back at about 60% memory. With the node healthy and nothing restarted in the application namespace, the event was treated as a one-off and left for Monday.
The next evening, at 6:02 pm, a different node raised the same SystemOOM event, again followed by a low EBS Burst Balance warning; the burst balance had been falling for about 2.5 hours beforehand. Pods again looked fine and node memory sat just under 70%. Datadog memory graphs then showed that the node's memory had been taken by the fluentd-sumologic pods and dropped sharply at the moment of the SystemOOM: Kubernetes had killed and restarted those pods to reclaim memory. The restarts had been missed the day before because the fluentd-sumologic pods ran in a separate namespace that nobody checked.
Symptom ? OOMKilled
Root cause ? Capacity exhaustion
Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.
When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.
same root cause · shared Telemetry agent, Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Telemetry agent
same root cause · shared Container memory limit