IL-0126 Meta AI · 2021-12-03 (published)ai-infra
Meta AI trained the 175B-parameter OPT model on GPU nodes rented from a cloud provider. In the two weeks after its 17 November 2021 update, the team restarted the run more than 40 times. The large majority of restarts were caused by hardware: nodes failed with ECC errors, and the team could not provision enough buffer nodes to replace a bad node quickly. A replacement through the cloud interface could take hours, often returned the same faulty machine, and sometimes arrived with NCCL or InfiniBand issues or the same ECC errors.
Recovery after each failure was slowed further by the checkpoint. A restart downloaded 1.6 TB (992 files of about 1.7 GB) from blob storage, and those downloads hung at random. Getting them to work reliably took considerable debugging of the provider's poorly documented Python API before the team settled on a solution it considered sufficient for the time being.
The team knew such failures would come at this scale but launched without automation, because it had to ship a trained 175B model by the end of H2 2021. Over the following two weeks it built its own tooling: a script to replace nodes, GPU burn-in tests to detect memory errors, automated InfiniBand tests, and monitoring of train.log. Working around infrastructure problems took up most of the team's time in that period, and the team set up an on-call rotation with runbooks for the most common issues.
Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.
Most restarts came from nodes that went down with ECC errors. The team could not get enough spare "buffer" nodes to swap in, and replacing one machine through the cloud provider's interface could take hours. More often than not the replacement turned out to be the same bad machine, and new nodes also came up with NCCL or InfiniBand problems or with the same ECC errors.
Each restart had to download the latest checkpoint, 1.6 TB in 992 files of about 1.7 GB each, from the cloud blob store, and these downloads started hanging nondeterministically, which delayed recovery further. The cloud provider's Python API for the blob store was poorly documented and needed a lot of debugging.
Symptom ? Training job failureGPU unavailable
Root cause ? Dependency failure
During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.
At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back.
The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.
same root cause · shared NCCL, InfiniBand fabric
same root cause · shared NCCL, InfiniBand fabric
same root cause · shared InfiniBand fabric
same root cause · shared InfiniBand fabric