IL-0122 OpenAI · 2025-07-15kubernetes

An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

An OpenAI engineer applied a configuration change with an invalid value to a core service. The same value was read by several services, which widened the impact, and it propagated quickly. New server instances failed on restart, pods entered crash loops, and error rates of ChatGPT conversations rose for a small share of users for about 55 minutes.

OpenAI reverted the value to the correct one and started a rolling restart of the affected pods to force them to resynchronise faster; full availability returned at 8:38 PM PDT.

Severity: –
Scope: ChatGPT conversations
Users affected: a small percentage of ChatGPT users
Region: –
Time to detect: –
Time to resolve: 55m

Symptoms

From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

New server instances of a core service failed whenever they restarted. As the invalid configuration value spread, pods went into crash loops.

Symptom ? Container crash loop High error rate

Root cause

Root cause ? Configuration change

Component ?–

Contributing factor ?Excessive blast radius No staged rollout

Trigger: An internal engineer applied a configuration change that contained an invalid value.

Detection

–

Remediation

reverted the configuration to the correct value
started a rolling restart of the affected pods to force them to resynchronise

Action items

Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops planned
Move to a service-specific configuration value instead of one shared between services planned
Improve deployment safeguards planned

Lessons

–

Sources

https://status.openai.com/incidents/01K08JJK77JF5D2NN636SDY9WD/write-up · official-status

Related cases

–