IL-0117 OpenAI · 2024-08-16networking
OpenAI attributes the incident to a combination of factors. A scheduled maintenance and an upgrade to the ingress of the user-facing clusters introduced a regression in the networking control plane, which caused a short outage of the data plane. Because of the momentary loss of connectivity, a set of services became unhealthy and were restarted automatically.
The restarts took much longer than expected. Each service ran a heavy query on first start, and so many services starting together overwhelmed a backend persistence store, which then needed extra time to catch up and recover.
Afterwards OpenAI mitigated the control plane regression and checked that control plane restarts no longer disturb the data plane, removed the expensive query from critical startup paths, and changed configuration so that the networking control plane affects the clusters' ability to handle traffic less.
On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.
Underneath, connectivity was lost for a moment, a set of services became unhealthy and were restarted automatically, and the restarts took much longer than expected because the starting services overwhelmed a backend persistence store with a heavy query they ran on first start.
Symptom ? High error rateDatabase overload
Root cause ? Breaking change in an upgrade
–
–
A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller