IL-0035 NU.nl · 2019-02-01 (published)kubernetesawsobservability
NU.nl ran a few Kubernetes clusters on AWS for logging and build tooling and some small internal customer-facing workloads. The team had been trying ElastAlert, which raises alerts from Elasticsearch logs. A new rule watched the Cloudfront-* indexes for missing documents. Because that Cloudfront distribution handles a few million requests per hour, the rule drove the ElastAlert pods' memory up steeply. The ElastAlert Helm chart has no default resource requests or limits, the team had not set any, and the namespace had no default limits, so the pods kept growing until the nodes' operating system ran out of memory, killed processes and left nodes NotReady. A Grafana dashboard of per-pod memory showed the ElastAlert pods climbing, and the ElastAlert problem was fixed. In hindsight, the team writes, it would have been better to use the ElastAlert options use_count_query and/or max_query_size.
The write-up then walks through the chain of failures around that cause. No alert fired, because no alerts on resource usage had been defined yet. Grafana and Prometheus ran inside the failing cluster, so the team was nearly blind; the same metrics were also in the EC2-based ELK stack, but not visualised well enough to be noticed. Logging, monitoring and customer workloads shared the same nodes, so one tool could exhaust resources for everything. The rule had passed code review, but it was installed from the command line by one team member, so others did not know it had been deployed and nothing recorded what had changed in the cluster. There was no smoke test after deploying that could have caught the memory growth or restarts. And knowledge of operating Kubernetes sat with part of the team; the on-call engineer had little infrastructure experience and needed help from colleagues.
The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.
When the team opened Grafana to investigate, Grafana and Prometheus showed the same erratic behaviour, because they ran on the same cluster. kubectl get nodes listed two worker nodes as NotReady (cluster version v1.10.6, nodes in eu-west-1), and several workers were only about two hours old.
kubectl describe node, on healthy and unhealthy nodes alike, showed the kubelet restarting and the node going NotReady and back to Ready, with the warning event "SystemOOM ... System OOM encountered" repeated four times. It looked as if the node's operating system was killing processes before the kubelet could reclaim memory itself.
The team first terminated the NotReady nodes one by one, expecting the auto-scaling group to bring up stable replacements. New nodes joined fine, but existing or new nodes soon went NotReady again. The one useful result was that Prometheus and Grafana landed on a node that stayed up, which gave the team the data to find the cause.
Symptom ? Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled
Root cause ? Configuration change
Users reported error pages, slow responses and timeouts from several services; no alert fired.
Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.
same root cause · shared Container memory limit
same root cause · shared kubelet
same root cause · shared Container memory limit
same root cause · shared kubelet