IL-0130 Google Cloud · 2024-10-22gcpai-infrakubernetes
Vertex AI Online Prediction serves models that customers deploy from Vertex Model Garden. For 13 days, 6 hours and 22 minutes, from 22 October 2024 at 14:08 to 4 November 2024 at 20:30 US/Pacific, deployments of large models, meaning models that need more than 100 GB of disk, failed in most regions with "Model server never became ready". The status updates tied the failures to deployments on L4 GPUs and on GKE Autopilot clusters.
Google's preliminary analysis placed the root cause in an internal storage provisioning configuration error that was implemented as part of a recent change. Engineers mitigated the problem by rolling that configuration change back, and the issue was resolved for all affected users as of 20:28 US/Pacific on 4 November 2024. Until then customers could work around it by deploying in one of the five unaffected regions.
From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error "Model server never became ready".
The failures happened in most regions. Deployments in asia-southeast1, europe-west4, us-central1, us-east1 and us-east4 were not affected. Status updates on 4 November described the failing deployments as large Model Garden deployments on L4 GPUs, and one update as large deployments (greater than 100GB) on a GKE Autopilot cluster.
The incident first appeared on the status page on 4 November 2024 at 13:17 US/Pacific, almost 13 days after it began, and the updates that day listed no workaround. The incident report published afterwards said customers had been able to deploy in one of the unaffected regions instead.
Symptom ? Deploys failing
Root cause ? Configuration change
The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.
–
A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.
–