IL-0059 NRE Labs · 2018-12-04kubernetesgcp
NRE Labs, the network automation lab site powered by the Antidote project and then in tech preview, ran on a single-master Kubernetes cluster built with kubeadm on GCP. kubeadm runs every master component, including a single-node etcd, as static pods managed by the kubelet, and gives etcd a liveness probe that reads the key foo with etcdctl.
On 4 December 2018 that probe began to fail although etcd was serving requests. With failureThreshold 8 and timeoutSeconds 15 the kubelet killed etcd about every two minutes, and the API server went down with it each time, so the site came back for a couple of minutes and then failed again. A kubelet restart looked like a fix for the same reason. The author then destroyed the whole cluster with terraform destroy and rebuilt it with Terraform and Ansible in about 30 minutes, which also destroyed the evidence; the problem returned on the new cluster.
Reading the container logs and the kubelet log showed that something outside etcd was sending it SIGTERM, and 8 × 15 seconds matched the two-minute cycle. The author removed the liveness probe from the kubeadm-generated etcd manifest and applied it between restarts, which ended the outage. Why the probe failed in the first place was never found.
The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`
On the master, a GCP virtual machine reachable over SSH, `docker ps` listed no API server container. Restarting the kubelet seemed to fix it: kubectl worked again, the syringe pod was restarted, a few lessons loaded, and an all-clear was posted. Shortly after, lesson activity dropped again.
`kubectl get pods -n=kube-system` showed etcd and kube-apiserver as Running, but each with 183 restarts, which the author did not notice at first. `kubectl get nodes` also showed one worker on v1.11.3 while the others ran v1.12.1, which the author put down to a version typo in one of his playbooks when he removed and added a node earlier that week.
Even on a freshly rebuilt cluster, `docker ps -a` on the controller showed etcd containers starting and exiting; etcd and the API server answered for about 1.5 to 2 minutes at a time. The etcd log showed a normal start, `embed: serving client requests on 127.0.0.1:2379`, and about two minutes later `pkg/osutil: received terminated signal, shutting down...`. The kubelet log said `Container {Name:etcd...truncated....} is dead, but RestartPolicy says that we should restart it.`
Symptom ? Control plane unresponsiveProbe failuresService outage
Root cause ? Not stated
The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.
Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.
–