IL-0121 OpenAI · 2025-07-09ai-infra

A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed

OpenAI deployed a faulty code change to a control plane cluster that mainly manages low-priority workloads. That cluster was also managing some production inference workloads, so the change degraded certain API model endpoints and reduced the availability of ChatGPT search. OpenAI rolled the change back to restore normal operation.

Afterwards OpenAI added safeguards that keep production inference workloads from being managed by that control plane cluster and strengthened testing for high-impact control plane changes. It is moving all existing production inference workloads under the production control plane cluster and building a circuit breaker meant to stop large-scale unintended changes.

Severity

–

Scope

gpt-4.1-mini API success rates degraded for about 20 minutes; ChatGPT search error rates raised for about 35 minutes

Users affected

–

Region

–

Time to detect

–

Time to resolve

55m