IL-0025 Imbue · 2024-06-25 (published)ai-infranetworking
This is one report of many problems found while building a training cluster, not a single incident. Hardware faults kept a tenth of hosts from booting. The top tier of the InfiniBand fabric had been wired as eight disconnected networks and had to be recabled, and recirculated exhaust air made switch ports run hot. Links that looked fine when idle flapped under load; the team burned in every port with synthetic traffic and automatically disabled links that the fabric manager flagged. Collective hangs came from driver lockups on single hosts and a threading misconfiguration, found with live debugging. A slow throughput sag turned out to be automatic garbage collection on one worker stalling a synchronous training step, fixed by scheduling collection on all hosts.
Roughly a tenth of new machines failed to boot. The fabric manager could not see all switches, ports ran hot, and under real traffic links flapped or threw errors. Collectives hung on every worker without a useful stack trace, and throughput sagged slowly during long runs.
Symptom ? Link flappingNCCL timeoutThroughput degradationGPU unavailable
Root cause ? Dependency failure
Trigger Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.
Automated hardware checks, fabric manager alerts, throughput monitoring and live debugging with profilers.
Verify physical wiring against the design, keep spare machines, test for every failure type you have met, and look for the host that did not log what the others did.
IL-0023 A training run failed on an NCCL timeout from one brief InfiniBand port dropsame root cause · shared PyTorch, NCCL, InfiniBand fabric
IL-0021 GKE GPU nodes came up without a working NVIDIA driversame root cause · shared GPU driver