IL-0141 Google Cloud · 2023-11-02kubernetesgcpobservability
gke-metadata-server is the GKE-managed component behind Workload Identity. Its versions 0.4.272 to 0.4.280 contained an incorrect configuration (the first detailed update called it a bug) that wrote debug logs containing "Unable to sync sandbox" at a high rate. Cloud Logging ingested those logs, so they consumed the customers' ingestion quota and created excess billable usage once the free monthly allotment was exceeded. The incident began at 10:45 US/Pacific on 2023-11-02, and the status page lists 39 Google Cloud locations as affected.
Google had identified the root cause by the 14:09 update on 2023-11-02 and rolled out a fix across the GKE fleet that stopped the excess logs from being ingested into Cloud Logging; on 2023-11-08 the rollout was about 25% complete. Until the fix reached a cluster, customers on the GKE Rapid channel could upgrade the cluster control plane to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, which carried a fixed gke-metadata-server. Customers on the Regular or Stable channels, or not using a release channel, had no workaround. The issue was resolved for all affected users as of 23:00 US/Pacific on 2023-11-09, and the incident was closed at 13:37 on 2023-11-10.
Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.
The logs came from gke-metadata-server, a GKE-managed system workload that runs as a DaemonSet in kube-system and is part of Workload Identity. Clusters running gke-metadata-server 0.4.272 to 0.4.280 were affected. To check a cluster, Google gave the command kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and the components.gke.io/component-version annotation in .spec.template.metadata.annotations: a version between 0.4.272 and 0.4.280 inclusive meant the cluster was affected.
The first detailed update, on 2023-11-02, said gke-metadata-server and the applications that depend on it kept working, but with some increased latency on metadata server calls and possibly reduced service, because gke-metadata-server was spending CPU on the excessive logging. From 2023-11-03 the updates said there was no impact on cluster functionality or performance and the only impact was extra logs in Cloud Logging.
Symptom ? Log floodHigh latency
Root cause ? Software defect
The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.
–
–
–