IL-0128 ByteDance · 2024-02-23 (published)ai-infranetworking
MegaScale is ByteDance's production system for training large language models on more than 10,000 GPUs. Its data center network is built from switches on Broadcom Tomahawk 4 chips in a three-layer CLOS-like topology. Each server has eight 200G NICs connected to eight different switches, and at the top-of-rack switches one 400G downlink port is split into two 200G ports with specific AOC cables. The paper lists frequent network interface flapping among the problems found and fixed in production training; it does not date individual events.
When an interface flapped it went down and back up, usually within several seconds, and dropped all packets in transit while down. With NCCL's default timeout, the collective communication timed out and returned a completion error before the network card recovered, so training stalled or slowed. The first lesson the team drew was to set the timeout threshold explicitly to a larger value. The second was that the root cause was poor link quality between the network card, the AOC cable and the switch; lower-level quality control over NIC signal strength, AOC cable quality and switch-side signal strength brought the flapping frequency down to a satisfactory level.
In its network tuning section the paper adds that NCCL's retransmit timer and retry count were tuned for fast recovery under link flapping, and that the adap_retrans feature was enabled on the NICs so that short flaps recover sooner through retransmission at shorter intervals.
ByteDance's large language model training jobs on its MegaScale system, which runs on more than 10,000 GPUs, occasionally stalled or slowed down. The cause was network interfaces that flapped frequently.
In a flap the network interface went down and then came up again, usually several seconds later, and every packet in transit while it was down was dropped. With the default timeout, NCCL timed out quickly and returned a completion error before the network card was back up.
Root cause ? DNS or network fault
MegaScale's monitoring system uses second-level monitoring to track link flapping and other NIC issues. When the RDMA traffic metrics carried in the training daemons' heartbeats drop sharply or fluctuate abnormally, the driver raises an alert for manual investigation, and if the traffic stops entirely it starts the fault recovery procedure automatically.
–
A default NCCL timeout can be shorter than the few seconds an interface needs to come back from a flap, so a brief link loss becomes a failed collective and a stalled job. A larger timeout and faster retransmission give the link time to return, but the flapping itself came from physical link quality between NIC, AOC cable and switch, and quality control at that level reduced it.
–