IL-0119 OpenAI · 2025-01-23kubernetesazure
ChatGPT depends on Cosmos DB from an external service provider. When Cosmos DB failed, chatgpt.com and a few APIs started returning errors. The provider recovered the affected databases at 4:25am PT, but errors went on for another 2 hours 43 minutes.
The second, longer part came from the web tier. The health checks of the web service pods failed, so the pods were marked unhealthy and crash-looped, and not enough pods were left to serve every request to the web layer. OpenAI at first attributed this to users retrying requests. Service recovered after OpenAI changed the web server's health checking logic, and the write-up gives 7:08am PT as the end of impact.
From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.
The errors had two causes in sequence. First, Cosmos DB, the database behind ChatGPT, failed; the service provider brought the affected databases back at 4:25am PT. Errors continued after that, because the ChatGPT web service pods were crash-looping: their health checks failed, the pods were marked unhealthy, and the pods that remained could not handle all requests reaching the web layer.
While this was happening, OpenAI believed the unhealthy pods were caused by users retrying their requests. The pods recovered only after a fix to the web server's health checking logic.
Root cause ? Cloud provider outage
–
The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database