IL-0125 BigScience · 2022-05-18ai-infra
BigScience trained a 176B-parameter multilingual model on 48 nodes (384 GPUs) of the Jean Zay cluster, as a SLURM job. A GPU died roughly every week. When the crash made the job exit, SLURM restarted it and at most about 3 hours of training were lost. Sometimes, though, a crashing process left the rest of the job hanging, and SLURM never learned that it should restart.
The team reduced the loss per crash by saving checkpoints more often. After a crash on 2022-03-21 cost 7.3 hours, the save interval went to 200 iterations, and on 2022-03-22, after an emergency maintenance downtime on Jean Zay lost some hours again, it went to SAVE_INTERVAL=100, at most about 3 hours of training. A checkpoint is 2.3TB and takes about 40 seconds to save, so saving too often has a cost. The hang was the bigger problem: Jean Zay gives no sudo access, and once one person starts a SLURM job nobody else can stop it, so a hang while that person slept or was away could not be handled until they returned.
The author reported the hang to PyTorch (issue 76287), where the suggested workaround was to set NCCL_ASYNC_ERROR_HANDLING=1, which forces a crash on NCCL problems such as a hanging broadcast, and to run srun with --wait=60 and --kill-on-bad-exit=1, so that a task exiting with a non-zero code ends the whole step. In a test on 2 nodes, killing one process by hand killed the local sub-processes, and about 5 minutes later SLURM killed the whole job. Jean Zay's SLURM turned out to have UnkillableStepTimeout=180, KillWait=60 and KillOnBadExit=1 already, so the only new setting was NCCL_ASYNC_ERROR_HANDLING=1. The NCCL part was not tested because a hanging NCCL operation is hard to emulate.
During BigScience's 176B-parameter multilingual training (tr11-176B-ml) on the Jean Zay cluster, launched on 2022-03-11, one of the GPUs died about once a week. Most of the time the SLURM job crashed and restarted automatically from the last checkpoint.
One such crash, at 2022-03-21_13:34:34 Jean Zay time, logged "terminate called after throwing an instance of 'c10::CUDAError'" and "what(): CUDA error: unknown error", raised from ../aten/src/ATen/cuda/CUDAEvent.h:95 in c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal(). It came 2 iterations before a checkpoint save, so 7.3 hours of work were lost.
Sometimes the training got stuck after a process crashed and did not quit, which the author connected to torch.launch and possibly NCCL. The job then sat in limbo without restarting until someone noticed, so someone had to watch the training around the clock. A stuck job could also break the team's kill switch, a file that, when created, makes the training save a checkpoint and exit gracefully.
Symptom ? Training job failureGPU unavailable
Root cause ? Hardware failure
There was no automatic detection of a hung job; it was noticed only when someone watching the training saw that it was stuck.
–
On a long run across hundreds of GPUs, a dead GPU every week is routine; what costs time is the case where a process crashes and the job hangs instead of exiting, because then nothing restarts it. Making NCCL errors fatal and having the scheduler end the whole step on a bad exit is meant to turn such a hang into a crash that the existing automatic restart handles; only the srun part was tested. Frequent checkpoints bound the loss per crash, at the price of save time for a 2.3TB checkpoint.
same root cause · shared NCCL, PyTorch