IL-0180 Sunwoo Park · 2023-05-18kubernetesgcpnetworking
The production GKE cluster ran Istio 1.7.4, released around late October to November 2020. Its code used the networking.k8s.io/v1beta1 API for Ingress and the admissionregistration.k8s.io/v1beta1 API for ValidatingWebhookConfiguration, and Istio 1.7 officially supported only Kubernetes 1.16, 1.17 and 1.18. Kubernetes 1.22 moved both APIs from v1beta1 to the GA version v1. When GKE auto-upgraded the control plane to 1.23 on 2023-05-18, istiod could no longer find the v1beta1 resources and Istio stopped working. Upgrading Istio from 1.7.4 to 1.13.3, whose code uses the v1 APIs, made it compatible with GKE 1.23 and resolved the outage.
The post works out why the failure came with 1.23 and not with the 1.22 upgrade in April. The reader found a GKE document saying that Ingress could keep using the v1beta1 API up to 1.23 as a one-time exception, and guessed that the webhook APIs had a similar grace period on GKE. The author then found that on GKE 1.23 and later the v1beta1 MutatingWebhookConfiguration and ValidatingWebhookConfiguration APIs are no longer available, which matches the timing. The reader also traced Istio's support history: up to 1.9 Istio used the v1beta1 APIs, 1.10 prepared for the deprecation and listed Kubernetes 1.22 as tested but not supported, and, by the reader's guess, 1.11 was the first to officially support 1.22. The reader suggested searching the Istio 1.7.4 code for the error "Patch webhook failed" to find where the deprecated API is used.
The original diagnosis, end of support for Istio, was corrected only because a reader questioned it by email; the cluster's upgrade history in Logs Explorer settled the timing.
On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.
In an earlier post the author had put the outage down to Istio 1.7.4 being out of support and fixed it by upgrading Istio. About half a year later a reader (called A in the post) wrote that end of support by itself should not change anything, pointed at the two missing v1beta1 APIs, and asked which GKE versions the cluster had run before and after the outage. A Logs Explorer query, resource.type="gke_cluster" protoPayload.metadata.operationType=~"(UPDATE_CLUSTER|UPGRADE_MASTER)" resource.labels.cluster_name="CLUSTER_NAME", showed an UPGRADE_MASTER operation from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18, nine minutes before the alert.
The control plane history for April and May read: 1.21.14-gke.18100 to 1.21.14-gke.18800 on April 2, 1.21.14-gke.18800 to 1.22.17-gke.5400 on April 11, 1.22.17-gke.5400 to 1.22.17-gke.6100 on April 15, 1.22.17-gke.6100 to 1.22.17-gke.7500 on May 13, and 1.22.17-gke.7500 to 1.23.17-gke.1700 on May 18. The outage followed the move to 1.23, not the move to 1.22 on April 11.
Symptom ? Service outage
Root cause ? Breaking change in an upgrade
A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.
–
The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer.
The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.
same root cause · shared GKE control plane
same root cause · shared Admission webhook
same root cause · shared Admission webhook