IL-0113 Webhook Relay · 2022-05-10kubernetesgcp
Webhook Relay has run on GKE since 2017, and the cluster in its main region holds the controllers and backend APIs. On 10 May 2022 that cluster's control plane stopped working properly. About 40 minutes into the outage the team traced it to the Kubernetes API server certificates: older GKE clusters were issued five-year certificates (newer ones get 30 years), and this cluster's certificates had never been rotated.
The team started a manual credential rotation with gcloud container clusters update --start-credential-rotation. The command succeeded but changed nothing: the nodes that the rotation should have recreated stayed as they were, so the node pools were recreated by hand. Upgrading the control plane did not help either. Workloads came back, but in-cluster service account authentication failed, so the pods could not reach Cloud SQL, Pub/Sub or Cloud Storage. GKE's multiple availability zones made no difference to this failure.
By 14:30 the team was already copying the main cluster's services into a new cluster. Load balancers and persistent volume claims still belonged to the old cluster, so the Kubernetes Service records were deleted there to free the load balancers for the new cluster, and the StatefulSet was detached from the old cluster as well. Webhook routes briefly wrote to a fresh disk; at 17:30 all writes were switched to the main disk, and at 18:30 the data from the temporary ClickHouse instance had been exported and inserted into the main one.
The team says the outage would have been much shorter had it known the managed cluster was beyond repair, because the cluster always seemed about to recover. Moving to a new cluster has its own work (moving persistent disks, detaching load balancers) but would still have been quicker.
At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.
Ten minutes in, everything the cluster reported looked inconsistent, as if the nodes were down or in the middle of an upgrade, although no maintenance window was scheduled. Recycling the node pools took a long time, the cluster timed out, and operations failed several times before they succeeded.
The GKE console neither looked clearly broken nor clearly healthy. Node pools could be listed, created and managed, deployments could be edited, logs could be viewed and the control plane version could be upgraded, but every operation was very slow. kubectl could list and fetch objects but could not update them, show logs or port-forward, and some of the workload information it returned was clearly stale.
After the nodes were recreated, workloads ran but service-account-based authentication inside the cluster failed. Backend services could not connect to Cloud SQL, Pub/Sub or Cloud Storage. A second cluster created during recovery also misbehaved: its control plane became unresponsive every few minutes. Port forwarding to the ClickHouse database that stores the webhook logs failed, and logs could not be read through the Kubernetes API either.
Symptom ? Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable
Root cause ? Certificate or credential expiry
The health check system sent a notification that the service was down; the team then found the main dashboard offline.
Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call.
On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.
same root cause · shared Node pool
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server