IL-0118 OpenAI · 2024-11-25kubernetesai-infra
A global change to Kubernetes namespace labels, without anyone intending it, made the networking layer recompute metadata. The small clusters handled this safely, but in three of the largest GPU clusters the recomputation overwhelmed the Kubernetes control plane. That led to cascading failures, high latency and elevated error rates across several OpenAI products.
OpenAI first reduced the impact by moving workloads out of the affected clusters by hand, which the unresponsive control planes made harder. After the root cause was found, OpenAI worked with its cloud provider to scale up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover. Some major workloads have since moved back into those clusters with additional protections.
From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.
ChatGPT was degraded for Paid and Enterprise users from 10:15 am to 12 pm, with error rates peaking at 13% for Paid and 23% for Enterprise; Free users saw degraded performance until 1:20 pm. Search in ChatGPT was disabled between 11:15 am and 12:05 pm. In the API, gpt-4-turbo-preview, gpt-4-o125-preview and text-embedding-3-large were hit hardest, with higher latency and error rates until as late as 1 pm.
The control planes of three of the largest GPU clusters were unresponsive, which made moving workloads out of those clusters by hand more complicated.
Symptom ? High error rateHigh latencyControl plane unresponsive
Root cause ? Configuration change
–
–
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server