IL-0040 Saltside · 2017-04-26 (published)kubernetesaws
Saltside runs several Kubernetes clusters. np.k8s.saltside.io is the nonproduction one, used for developer experiments and as the target of CI builds. The CI process builds the application's Helm charts for four markets, installs them on this cluster and runs helm test on each release. Running it collapsed the cluster's three t2.medium workers: pods went Pending and then Unknown, image pulls failed, Tiller crash-looped, deletes had no effect, and the nodes reported the kubelet or the container runtime as down. The SRE lead ended it by terminating the three workers and letting the auto scaling group replace them; the cluster was normal again after about 10 minutes. The post notes that this cannot be the standard fix, because pods are not drained to other nodes first.
To find the bottleneck, the team fixed the Datadog integration for the nonproduction AWS account and repeated the installs while watching EBS metrics. The EBS queue length shot up. kops had created the nodes with 20GB gp2 root volumes by default, gp2 IOPS grow with volume size, and df showed 90 percent of the disk in use. kops also did not support EBS optimization. With the root volumes raised to 120GB, and a DaemonSet that ran a sleep loop in common base images such as ruby:2.4 and alpine:3.2 so that they stayed pulled on every node, the charts installed without killing the kubelet. Other things still broke, so the cluster could still not handle that many container operations in parallel.
Other findings from the review: Tiller failed to reschedule because of CPU limits although the nodes showed no CPU request problem, which still needs investigation; helm init runs Tiller as a single replica, so a crash-looping Tiller blocks the chart installs needed to deploy fixes; there was no way to throttle how many new pods are created; the Datadog agent did not yet support the pod phase metrics from kube-state-metrics; and the team did not know what the kubelet checks such as "Disk Pressure" test.
While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.
kubectl describe pod showed pod events full of image pull errors from Docker. These were first put down to overload: the cluster was nonproduction, the nodes were t2.medium, and pulls from the official Docker registry had often failed before.
Clean-up did not work. Tiller, Helm's server-side component in kube-system, was in a crash loop, so Helm releases could not be deleted. Plain kubectl delete commands returned success but removed nothing. With --force --grace-period=0 the pods disappeared from kubectl get, but nobody checked whether the containers were gone in Docker. In kube-system some pods showed DeadNode. kubectl describe nodes showed some nodes reporting the container runtime down and others the kubelet down after failed checks, kubectl get nodes listed all three workers as not ready, and SSH connections to the instances timed out.
No monitoring reported any of it. There were no node CPU, memory or disk metrics, no metrics on pod phases such as Pending or Unknown, and no CloudWatch metrics in Datadog for this AWS account.
Symptom ? Pods PendingImage pull failuresNode NotReadyService outage
Root cause ? Capacity exhaustion
The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.
An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.
same root cause · shared Node root disk, Container runtime, kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet