IL-0039 FREE NOW · 2019-09-19kubernetesaws
FREE NOW runs its Kubernetes clusters on AWS with kops, and the worker autoscaling groups in every environment use spot instances. Nodes are expected to disappear at any time, and the clusters cope as long as replacements eventually join. When a new worker boots, kops nodeup installs docker-ce, which depends on the container-selinux package; nodeup downloaded that package from a hardcoded URL on a CentOS mirror (kops issue 7608). Two or three days before the incident the package was moved inside the mirror. From then on every new worker failed to bootstrap and never joined, and as spot nodes were terminated the clusters slowly ran short of capacity.
On 2019-09-19 the Prelive cluster ran out of room: pods stayed Pending and deployments failed. By 16:20 the team had matched it to the kops issue and realised that capacity could vanish from every cluster without being replaced. At 16:40 the on-call team was warned that production could become badly impaired at any moment, and at 16:55 developers were asked not to deploy to production. Downloading the package by hand on Prelive nodes let them finish bootstrapping and join.
There was no fixed kops release. The first workaround uploaded the package to a public S3 bucket and added a kops startup hook to fetch it during bootstrap. It failed first on the syntax of the systemd unit and then, at 18:20, because startup hooks run too late in the boot. The second workaround switched the cluster AMI from Amazon Linux 2 (CentOS) to Debian. It worked on Prelive at 18:55, was applied to all clusters at 19:00, and the incident was declared resolved at 19:08.
At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.
Pods could not be scheduled because the cluster had no free capacity, so they stayed Pending and deployments failed. The Prelive autoscaling group held four fairly new instances that had never joined the cluster. Logging in to one of them over SSH showed that the container-selinux package could not be installed because of a checksum mismatch, so the node's bootstrap never finished.
The mismatch had been building for a while: on some clusters the number of worker nodes in the autoscaling groups had differed from the number of nodes registered in Kubernetes for a long period without anyone noticing. Because the workers are spot instances that can be terminated at any moment, every cluster could lose capacity that would not come back.
Symptom ? Pods PendingScale-up failure
Root cause ? Dependency failure
Developers asked for help in the #ask-platform Slack channel because their deployments were failing.
Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity.
What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.
same root cause · shared Node operating system image, Node pool
same root cause · shared Node operating system image