IL-0132 Google Cloud · 2022-06-02gcpai-infra
Google's preliminary analysis found that a faulty resource cleanup process marked Vertex Prediction Endpoints as deleted across all regions. With the endpoints marked deleted, every pre-existing model deployed on them was undeployed, so online prediction requests failed and customers could not simply redeploy because of quota failures. The service recovered fully once the endpoints were restored.
The mini incident report gives the incident as 10:10 to 14:30 US/Pacific on 2 June 2022, 4 hours 20 minutes. The last status update says the issue was resolved for all users as of 15:21 US/Pacific. The report says the information was preliminary.
From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.
Customers found that all their existing models had been undeployed from their Vertex AI Endpoints. Listing the deployed models of an endpoint returned an empty list, Predict and Explain requests failed with runtime exceptions and general errors, and trying to deploy the models again failed with quota errors. Google offered no workaround while the incident lasted.
The status page entries listed 19 affected regions across Asia, Australia, Europe and North America.
Root cause ? Software defect
The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.
–
–
same root cause · shared Model request routing
same root cause · shared Model request routing