IL-0075 Civis Analytics · 2019-01-16 (published)kubernetesawsobservability
Civis Analytics runs its Data Science Platform on Kubernetes: Jupyter notebooks, client services and, since autumn 2018, client scripts. When script workloads moved to Kubernetes, the cluster grew to about 290 EC2 instances and the three API servers, running on m4.large masters with no CPU limit, saturated CPU and memory and made the control plane unusable. Moving internal users back to the old infrastructure stabilised the cluster. The author names the main mistake as assuming that Kubernetes handles scale for free.
The team set a goal of 500 nodes, with 1,000 as a stretch, and ran load tests on a staging cluster by raising the size of the AWS autoscaling group. They added Prometheus through the CoreOS Prometheus operator for API metrics. Larger masters (three m4.4xlarge, 16 CPU and 64 GiB each, against the Kubernetes recommendation of c4.4xlarge for 251 to 500 nodes) helped, but the cluster still struggled before 500 nodes.
Load tests without the Datadog and Sumologic (fluentd-based) DaemonSets reached about 800 nodes before latency and 5xx errors rose; adding Sumologic back still reached more than 700, which pointed at Datadog. With DD_KUBERNETES_COLLECT_METADATA_TAGS, every Datadog agent, one per node, polled the API server for the names of all services so it could tag pod metrics, and those tags were not even useful to Civis. Datadog offers two fixes: turning that setting off, or running the Datadog Cluster Agent so only one agent talks to the API server. With the Datadog changes the cluster reached more than 700 EC2 instances easily, and Civis concluded it could support 700 nodes.
Two other changes stayed in place without making a large difference: max-requests-inflight for non-mutating requests went from 400 to 1200, which reduced 429 responses, and the etcd EBS volumes grew from 20 to 100 GB, raising IOPS from 100 to 300. The team also found kube2iam running out of memory because it kept every pod's metadata in memory, and notes the same pattern in the fluentd Kubernetes metadata filter. For the later DNS timeouts, scripts were switched from the ClusterFirst DNS policy to Default, which skips cluster DNS.
After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.
The serious failures came when the cluster was loaded with a large number of scripts and grew to about 290 EC2 instances; it had never run more than 80 before. CPU on the master nodes went high, and the three Kubernetes API server containers were pegged on CPU. The API servers had no CPU limit, so they starved the other control plane components on the same nodes and the cluster became unusable. The API servers also used up all the memory on their node before restarting.
The team could see master-node CPU in CloudWatch and per-container CPU and memory through Datadog agents, but not API request throughput, 5xx error rates or request latency. Once they deployed Prometheus, their load tests showed API response latency and 5xx error rates spiking at peak load, and the API server access logs contained several 429 responses.
After the infrastructure was released again, users reported scripts failing with DNS timeout errors. The team could not reproduce them reliably and only connected them to the larger cluster size.
Symptom ? Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures
Root cause ? Capacity exhaustion
Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.
Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.
same root cause · shared Telemetry agent, Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Telemetry agent
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server