A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API

Symptoms

At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

By 01:45 AM PDT on 10 June errors stood at about 5 percent while capacity was rebalanced. The worst period was from 2:00 AM to 8:00 AM PDT: ChatGPT error rates peaked at about 35 percent, and API availability dropped to 75 percent, with API error rates peaking at about 25 percent.

Most major API models were fully working again by 9:40 AM PDT, the API was fully recovered at 12:30 PM, and all affected systems were restored by 3:00 PM PDT.

Symptom ? High error rateGPU unavailable

Root cause

Trigger

A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

Detection

Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

Remediation

• re-imaged the affected GPU virtual machines
• deployed recovery automation to restart the remaining impacted nodes
• halted the background update mechanisms on the GPU VMs

Action items

• Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way done
• Update system configurations to prevent conflicts between systemd-networkd and the networking service done
• Audit VM configurations across the fleet to find and reduce similar risks planned
• Improve recovery speed for critical infrastructure such as GPU VMs and clusters planned
• Run regular disaster recovery drills planned

Lessons

An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

Sources

• https://status.openai.com/incidents/01JXCAW3K3JAE0EP56AEZ7CBG3/write-up · official-status

Related cases

• IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

  same root cause · shared systemd-networkd, Node operating system image

• IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025

  same root cause · shared Node operating system image

• IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

  same root cause · shared Node operating system image