IL-0116 OpenAI · 2023-01-25kubernetesnetworkingai-infra
OpenAI uses all the GPU capacity it can get, so its clusters run on several hardware and networking configurations, and each needs its own CNI configuration. A change to that configuration was fully tested in staging, but staging lacked one hardware variant that exists only in production. The change was incompatible with servers of that variant, and their workloads lost network connectivity when the change reached the GPU cluster that had them. At 4:53 pm Pacific on 25 January 2023 this became a major outage of that GPU cluster and a full outage of the text-davinci-003 model.
It took almost an hour to trace the connectivity loss to the CNI change. The same change had been deployed to other clusters over the previous 24 hours and had been deemed safe. Once a fix was in place, 80% of traffic returned immediately. Bringing the rest back took about three more hours, because known network and storage bottlenecks limit how many models can be loaded onto GPUs at once and because faulty hardware had to be identified and removed.
At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.
Engineers saw at once that workloads on the cluster had lost network connectivity, but it took nearly an hour to find a CNI configuration change as the cause. The change had been deployed to other clusters over the previous 24 hours and was considered safe.
After the fix, 80% of capacity came back within about an hour of the start. Reliability stayed impaired for about three more hours while models were deployed back onto GPUs: known bottlenecks in network and storage infrastructure limited how many models could be loaded at a time, and misbehaving hardware had to be found and taken out of operation. Capacity was fully stable after about four hours.
Symptom ? Service outageThroughput degradation
Root cause ? Configuration change
Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.
A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.
–