IL-0131 Google Cloud · 2024-06-12gcpai-infra
Beginning 8 June 2024, a new form of request to the Vertex AI GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers. A server that crashed restarted, and load balancers sent user queries to other healthy servers until it was back in service, so users saw nothing at first. Google's internal production monitoring alerted engineers on 10 June at 07:35 US/Pacific; they found the cause of the segfault, developed a fix and began rolling it out progressively. The earlier mini incident report described the cause as a bug in a recent change to the Vertex AI online prediction platform that affected the processing of generative requests.
On 12 June at 12:06 the rate of segfault-triggering traffic increased, and enough servers were offline at the same time that the remaining capacity could no longer serve the load. Vertex AI and the products built on it, Dialogflow CX, Agent Assist and Contact Center AI, returned errors in eight regions. Engineers quickly confirmed it was the same issue found on 10 June and accelerated the rollout already in progress. The rollout completed at 15:59 and fully mitigated the issue after 3 hours 53 minutes.
From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.
Vertex AI Online Prediction showed high latency and elevated 500 and 502 error rates on the Predict, RawPredict, GenerateContent and StreamGenerateContent methods, and some prediction requests failed with "CANCELLED" errors. Dialogflow CX Generators, Generative Fallback and ML entities returned elevated "INTERNAL" and "DEADLINE_EXCEEDED" errors and in some cases timed out. Agent Assist, including (Proactive) Generative Knowledge Assist, and Contact Center AI Insights had elevated error rates in LLM summarization and topic modelling. Some BigQuery generative features also had elevated error rates.
The status page offered no workaround. At 16:20 Google reported notable recovery and expected us-central1 to be fully mitigated within the hour, with no estimate for other regions; at 17:21 it declared the issue resolved for all affected projects.
Root cause ? Software defect
Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.
A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once.
Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer