IL-0183 ckck803 · 2026-02-22kubernetes
A cluster built with kubeadm uses TLS certificates that kubeadm generates for internal communication. The CA certificates are valid for 10 years, but the leaf certificates are valid for one year by default and must be renewed every year. On this cluster all eleven leaf certificates expired at 2026-02-20 16:16 UTC. kube-apiserver could no longer complete the TLS handshake with etcd and exited every time it started, and without an API server the kubelet could not register its node and Calico could not tear down pod networks.
The author renewed all certificates with sudo kubeadm certs renew all. Because the cluster was completely down, kubeadm printed "Error reading configuration from the Cluster. Falling back to default configuration" and renewed with its defaults. kube-apiserver, kube-controller-manager, kube-scheduler and etcd run as static pods, so restarting containerd and the kubelet restarted them with the new certificates. The renewed /etc/kubernetes/admin.conf was then copied to $HOME/.kube/config, and commands such as kubectl get nodes were checked to work.
The cluster could not be reached from outside. On the control plane node, sudo systemctl status kubelet showed TLS handshake errors such as "http: TLS handshake error from 101.198.0.183:19517: tls: client requested unsupported application protocols" and "http: TLS handshake error from 101.198.0.172:36328: tls: client offered only unsupported versions: [302 301]".
The kubelet kept failing to register its node, "Unable to register node with API server" err="Post "https://x.x.x.x:6443/api/v1/nodes": dial tcp x.x.x.x:6443: ...", and to renew its node lease, "Failed to ensure lease exists, will retry". containerd logged that the Calico CNI plugin could not tear down pod sandboxes because it could not reach the API server: "plugin type="calico" failed (delete): error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: connect: no route to host".
sudo crictl ps -a showed kube-apiserver containers in state Exited with attempt counts that kept rising (7, 8 and 41): the API server died right after it started. The log of one of these containers showed only "oidc-issuer-url and oidc-client-id must be specified together when any oidc-* flags are set", which did not point to certificates. A more recent container's log held the real error, from kube-apiserver's connection to etcd at 127.0.0.1:2379: "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-02-22T06:12:12Z is after 2026-02-20T16:16:45Z".
sudo kubeadm certs check-expiration listed admin.conf, apiserver, apiserver-etcd-client, apiserver-kubelet-client, controller-manager.conf, etcd-healthcheck-client, etcd-peer, etcd-server, front-proxy-client, scheduler.conf and super-admin.conf as expiring Feb 20, 2026 16:16 UTC, with residual time <invalid>. The CA certificates ca, etcd-ca and front-proxy-ca were valid until Feb 18, 2035.
Root cause ? Certificate or credential expiry
The author could not connect to the cluster from outside and went to the control plane node to read the kubelet logs.
–
A kubeadm cluster stops working one year after its leaf certificates were issued unless they are renewed. The first errors seen may be TLS handshake failures or a flag error that does not mention certificates; the decisive message is the x509 expiry in the kube-apiserver log for its connection to etcd.
The author recommends checking expiry with kubeadm certs check-expiration and renewing from 30 days before. kubeadm upgrade apply renews the certificates too, so a cluster upgraded at least once a year avoids the problem; a cluster run for a long time without upgrades can renew from cron, for example with 0 3 1 1 * root kubeadm certs renew all && systemctl restart kubelet in /etc/cron.d/kubeadm-cert-renew.
same root cause · shared kubelet, Kubernetes API server
same root cause · shared Kubernetes API server