IL-0184 dbhang · 2021-03-31 (published)kubernetes
A kubeadm-built cluster that runs JupyterHub stopped accepting requests because its internal certificates, including the API server certificate, had expired. The author had not used kubeadm since building the cluster and had not known that the cluster relies on certificates that expire.
With kubectl unusable, the certificates were renewed with kubeadm on the control plane node. The author checked the remaining validity with kubeadm alpha certs check-expiration, moved the old certificates and the /etc/kubernetes/*.conf files to a backup directory, regenerated all certificates with kubeadm init phase certs all and the kubeconfig files with kubeadm init phase kubeconfig all. kubeconfig files such as admin.conf have to be regenerated separately to match the new certificates; the output shown came from a kubeadm client at v1.16.3.
Copying the new admin.conf to ~/.kube/config and restarting kubelet was not enough: the kube-system pods still held the old credentials and kubectl answered Unauthorized. Restarting all of those containers with docker restart made kubectl work again and pods could be created. Containers without an automatic restart policy have to be started by hand. The author notes that anyone who had built a cluster from scratch would know about these certificates, and that a single command would have prevented the outage.
KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.
The log read: HTTPSConnectionPool(host=hostVM, port=6443): Max retries exceeded with url: /api/v1/namespaces (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))). Searching for it showed that the certificates used for communication inside the Kubernetes cluster had expired.
Every kubectl command, including the ones meant to renew certificates, failed with "The connection to the server hostvm:6443 was refused - did you specify the right host or port?", because the Kubernetes API server certificate had expired too. After new certificates were generated and admin.conf was copied into place, kubectl returned "error: You must be logged in to the server (Unauthorized)" until the kube-system containers were restarted.
Root cause ? Certificate or credential expiry
KubeSpawner failed to create a container and logged an SSL certificate verification error.
–
kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.
same root cause · shared kubelet, Kubernetes API server
same root cause · shared Kubernetes API server