IL-0127 Meta · 2024-07-23 (published)ai-infra
Meta trained Llama 3 405B on up to 16K H100 GPUs, each at 700W TDP with 80GB of HBM3, on Grand Teton servers with eight GPUs joined by NVLink and two CPUs per server. Jobs were scheduled by MAST, the large model used a RoCE network fabric, and collectives ran on NCCLX, Meta's fork of NCCL. The paper says the failure modes of 16K-GPU training exceed those of the much larger CPU clusters Meta runs, and synchronous training makes the job less tolerant of faults.
In a 54-day snapshot the job stopped 419 times without warning. About 78% of those stops came from confirmed hardware faults, such as failed GPUs or host components, or from suspected hardware problems such as silent data corruption and unplanned maintenance of a single host. Faulty GPUs (148) and GPU HBM3 memory (72) were the largest categories, followed by software bugs (54) and network switches or cables (35). Automated cluster maintenance, such as firmware and Linux kernel upgrades, added at least one interruption a day on top of this.
Automation handled almost all of these interruptions; significant manual intervention was needed only three times in the period, and effective training time, the share of elapsed time spent on useful training, stayed above 90%. Meta shortened job startup and checkpointing, used the PyTorch NCCL flight recorder to record every collective and dump traces on NCCLX watchdog or heartbeat timeouts, and built NCCLX together with PyTorch so a stalled collective is detected, timed out and snapshotted with its pending transfers between ranks.
Over a 54-day snapshot of Llama 3 405B pre-training the training job was interrupted 466 times. 47 interruptions were planned: automated maintenance such as firmware upgrades, or operator actions such as configuration or dataset updates. The other 419 were unexpected. Because training is synchronous, a failure on a single GPU can force a restart of the whole job.
The paper's breakdown of the 419 unexpected interruptions, with its own counts and percentages: faulty GPU 148 (30.1%), GPU HBM3 memory 72 (17.2%), software bug 54 (12.9%), network switch or cable 35 (8.4%), unplanned host maintenance 32 (7.6%), GPU SRAM memory 19 (4.5%), GPU system processor 17 (4.1%), NIC 7 (1.7%), NCCL watchdog timeouts 7 (1.7%, cause unknown), silent data corruption 6 (1.4%), GPU thermal interface and sensor 6 (1.4%), SSD 3 (0.7%), power supply 3 (0.7%), server chassis 2 (0.5%), IO expansion board 2 (0.5%), dependency 2 (0.5%), CPU 2 (0.5%) and system memory 2 (0.5%). GPU issues as a group account for 58.7%, and about 78% are confirmed or suspected hardware problems.
Some failures give no clear error. When a remote GPU or an NVLink connection fails, the failure often shows up only as stalled load or store operations inside CUDA kernels, without a clear error code. Such stalls cannot be completely prevented; Meta's system monitors the state of the communication library and times out automatically when it detects one.
Root cause ? Hardware failure
The communication library's state is monitored and a stalled collective is timed out automatically; on an NCCLX watchdog or heartbeat timeout the PyTorch NCCL flight recorder dumps its tracing data, including collective metadata and stack traces.
Training on 16K GPUs has more failure modes than Meta's much larger CPU clusters, and because training is synchronous a single GPU failure can force a restart of the whole job. Meta raised effective training time by cutting job startup and checkpoint time and by building tools for fast diagnosis, and kept it above 90% while automated firmware and kernel maintenance caused at least one interruption a day.
same root cause · shared PyTorch, NCCL