IL-0123 OpenAI · 2023-11-08ai-infra
OpenAI's routing layer had a chronic memory problem: its code allocated a new responseBuffer on every pass of a loop instead of reusing one, and garbage collection fell behind, especially under many requests. On a morning with dramatically more completions than any earlier day, routing nodes reached their memory limits and failed readiness checks, until too few nodes were left to carry the traffic and the service could not recover.
OpenAI ended the outage by limiting incoming traffic, redeploying the service en masse and then ramping traffic back up slowly. Afterwards the code was changed to pre-allocate the buffer and reuse it, which brought a 3X improvement in both memory and CPU usage. The memory limits were set to an appropriate level with significant headroom, rate limit controls were added to shed load more gracefully, and capacity was increased. Autoscaling had not been possible for this service because adding capacity could hurt an upstream service; that issue has been fixed and autoscaling is to be configured.
Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.
Nodes of the routing layer hit their memory limits and failed readiness checks. As more of them became unavailable, the remaining nodes did not have the capacity to serve incoming traffic, and the service could not recover on its own. That morning had dramatically more completions than any day before.
Root cause ? Software defect
–
–
same root cause · shared Model request routing
same root cause · shared Model request routing
same root cause · shared Container memory limit