IL-0140 Google Cloud · 2022-06-14kubernetesgcp
For 3 hours and 19 minutes the GKE API in us-central1 had increased latency and errors. Google's preliminary analysis attributes it to an unexpected increase in backend traffic, made worse by clients retrying failed requests. Engineering mitigated it by scaling up the GKE control plane to handle the extra traffic, and advised customers who could use another region to do so meanwhile.
Running workloads were not affected; the impact was on cluster management operations through the API and on Cloud Data Fusion and Cloud Logging features that depend on it. The status page gives 06:22 as the incident end and 06:17 as the time the issue was resolved for all affected users.
From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.
Cluster workloads kept running, but customers could not reliably check cluster status, start upgrades, view workloads, or create and delete clusters and node pools. Cloud Data Fusion instance creation and deletion failed in us-central1, and suggested searches for GKE clusters in Cloud Logging were slow.
Root cause ? Capacity exhaustion
–
–
Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.
–