IL-0133 Nebius · 2026-08-19kubernetesai-infra
Around 08:30 UTC a storm-related event at the building hosting us-central1 took the building management system offline and shut down the chilled-water cooling loop. The building management system is also what monitors the facility, so neither the facility operator nor Nebius received a facility-level alert. The data halls overheated within about two hours, and servers, network switches and rack power systems shut down on thermal protection. By the time a controlled reduction of heat load was considered, about a quarter of the affected servers had already shut down. Nebius reached the facility operator at 10:46, cooling was switched to manual bypass at 11:15 with rack rear doors opened, temperatures began to fall at 11:45, and tripped racks were re-energised from 12:11, control-plane racks first. The facility was thermally stable about six hours after the event.
Restoring the platform took much longer than restoring the facility. The regional infrastructure control plane did not come back on its own after power returned and needed about two hours of manual work, until 12:42. Automatic VM recovery makes a single attempt per instance and is rate-limited to protect the platform during isolated failures. It was launched for about a third of all running instances in the region; about a third of those attempts succeeded, and most of the rest failed on stale disk attachment state left on hosts that had lost power abruptly. Block-storage host components were restarted between 13:41 and 13:56 to clear the stale state, but many earlier attempts had already failed and were not retried automatically, and the recovery rate limits were raised by hand around 14:25.
Managed Kubernetes stayed down longest because of a bug in Compute disk hot-plug: when an instance was recovered, a secondary data disk could be left unattached. Managed Kubernetes control planes keep their state on such disks, so hundreds of recovered control-plane instances came up without their data disk. The bug was identified at 18:17, a controlled stop and start of affected control-plane instances was tested on one cluster by 18:30, and it was applied at scale between 19:36 and 19:55, which cut the clusters with an unreachable control plane from 55 to 8. All Managed Kubernetes clusters were reported healthy at 21:58. The incident was fully mitigated at 08:38 UTC on 20 August, when all reserved GPU capacity was confirmed restored; repair of the remaining on-demand nodes went on for several more days. The hot-plug bug has been fixed and the fix deployed to all regions.
On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.
Regional dashboards had shown the temperature rising from about 09:15, but no alert existed on that metric, and the rise was only noticed in retrospect. Between about 10:40 and 11:00 rack power systems tripped on thermal overload and took down racks that hosted regional control-plane and network gateway components; peak air inlet temperature reached 58.5 °C. VMs lost external connectivity from about 11:00 to 12:50, requests to the regional S3 endpoint failed from about 10:40 to 13:27, block and file storage operations failed or hung, Token Factory models were partly or fully unavailable from about 10:37, and the regional console and API endpoints were down. Customer-facing metrics and logs for regional resources were missing or incomplete, and the responders' own monitoring was weakened because it depended on components inside the region. At 11:01 a health check wrongly reported the region as recovered and sent part of the console traffic back to it for about seven minutes.
Managed Kubernetes user cluster control planes became unavailable across the region. After power returned, the first recovery wave between 15:43 and 16:05 brought about half of them back; the rest either failed recovery or came back unhealthy. Around 16:15 about 140 clusters were estimated unreachable. Starting control-plane instances by hand restored about another quarter of the affected clusters by around 17:00, but at 18:11 79 clusters still had an unreachable control plane with their etcd data unavailable.
Symptom ? Control plane unresponsiveService outageMonitoring outage
Root cause ? Dependency failure
No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.
Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.
same root cause · shared Data center power and cooling, InfiniBand fabric
same root cause · shared InfiniBand fabric
same root cause · shared InfiniBand fabric
same root cause · shared Data center power and cooling
same root cause · shared InfiniBand fabric