IL-0134 Lambda · 2026-02-14ai-infranetworking
A power event at Lambda's AUS01 data center cut power and rebooted 246 GPU nodes, all 4 spine switches, the mCPU nodes and the VAST storage infrastructure at 21:06 UTC on 14 February 2026. The GPU nodes came back quickly, but storage had disconnected and customer workloads failed.
Engineers restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back, and restored part of the UPS redundancy while work continued on storage and protected power. They then confirmed that all nodes in AUS01 were powered up with network connectivity, completed physical checks, and moved on to the logical startup of storage, which NFS and the services that depend on it need. Starbase and the other storage-dependent services in us-west-3 were then restored.
When the incident was marked resolved, the main remaining problem was VMs with stale or empty NFS mounts. Lambda told affected customers to reboot their VMs to get file system access back, and to reboot the hypervisor if a VM reboot was not enough.
At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.
The power loss had rebooted 246 GPU nodes, all 4 spine switches, the mCPU nodes and the VAST storage infrastructure. Lambda restored the compute fabric and InfiniBand connectivity first and started the storage again later.
After storage and storage-dependent services were back, some customer VMs still had stale or empty NFS mounts left over from the power event and the mass reboots.
Symptom ? GPU unavailableStorage disconnected or stale mounts
Root cause ? Dependency failure
–
–
A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.
same root cause · shared InfiniBand fabric, Data center power and cooling
same root cause · shared InfiniBand fabric
same root cause · shared InfiniBand fabric
same root cause · shared Data center power and cooling
same root cause · shared InfiniBand fabric