IL-0135 Crusoe Cloud · 2023-11-15ai-infra
Crusoe Cloud was carrying out routine maintenance on the UPS serving one row of servers in a us-northcentral1 data center. To allow work on the UPS without interrupting power to the row, the UPS was put into bypass mode. During the maintenance window, power switched from the primary source to a backup source for unrelated reasons. With the UPS bypassed, nothing regulated the switchover, so the row lost power and all its servers shut down, stopping a percentage of the region's VMs.
Teams identified the affected VMs and restarted them as soon as power was restored. Service came back in stages, and the final VM was restarted at 00:20 UTC on 16 November 2023, 3 hours 49 minutes after the power failure. Crusoe Cloud said its future data centers will have redundant UPSs so this situation cannot recur, and that it is reviewing its UPS maintenance procedures.
At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.
The status page reported certain VMs in us-northcentral1-a as unavailable at 21:07 UTC. The team identified the affected VMs and started restarting them once power was back; by 23:05 UTC most were running again, and the last VM was restarted at 00:20 UTC on 16 November.
Symptom ? GPU unavailableService outage
Root cause ? Dependency failure
The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.
When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.
same root cause · shared Data center power and cooling
same root cause · shared Data center power and cooling