IL-0115 OpenAI · 2022-06-23kubernetesazureai-infra
OpenAI ran its models on three production clusters on Azure Kubernetes Service (AKS), managed with Terraform and Kubernetes. One of them went completely offline through a mix of human error, holes in the safeguards and subtly missing information about what destructive operations would affect. The models it served were moved onto the remaining two clusters, and most of them recovered before the lost cluster came back.
Fine-tuned models took much longer. Losing a whole cluster used up the usual headroom, GPUs were the most constrained resource, and supply chain limits made it impossible to add GPUs at a few hours' notice. Each fine-tuned model is very large and normally only a few are loaded at a time from Azure Blob Storage; restoring all of them from scratch was bottlenecked by Blob Storage bandwidth, made worse by cross-cluster transfer, and the move invalidated the layers of caching that normally hide this. Fine-tuned Davinci did not fully recover until the cluster itself was back.
Rebuilding the cluster, normally fairly quick, took until about 05:30 PDT, around 10½ hours. During setup the connection to AKS was lost; with Azure support the VPN link was rebuilt. Later the connection to Azure Private Link was lost too and workarounds were built, again with Azure support; the cause of both was still unknown at the time of writing. A Kubernetes upgrade was expected to ease some of the network problems, but after the upgrade the ingress turned out to be incompatible with the new Kubernetes version, fixing it was not feasible in the time available, and AKS does not allow a Kubernetes downgrade. The team finally tore the cluster down, rebuilt it from scratch, and moved traffic back between 05:30 and 06:30 PDT.
At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.
Codex, Embeddings and the legacy GPT models were completely unavailable for about an hour. Most requests to fine-tuned Curie timed out until 00:10 PDT, about five hours, and most requests to fine-tuned Davinci timed out until 06:40 PDT, about twelve hours.
Symptom ? Service outageHigh error rateIntermittent timeouts
Root cause ? Operator error
–
Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild.
Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller