IL-0023 Databricks · 2026-07-01 (published)ai-infra
The port was down longer than NCCL's InfiniBand transport timeout, about seven seconds with retries, so the connection was torn down before the PyTorch watchdog could react. The fleet health check counted port flaps rather than cumulative downtime, and one flap stayed under its threshold. The team raised the NCCL timeout default, switched the check to cumulative downtime, and made jobs restart from a checkpoint instead of leaving GPUs idle until the watchdog fires.
About seven hours into a training run the job died with an NCCL timeout. The logs gave no cause.
Root cause ? Dependency failure
Trigger One InfiniBand port used for RDMA collectives went down once and came back.
The job crashed with the timeout; the continuous health checks had not flagged the single event.
Cumulative downtime matters more than the number of flaps, the watchdog timeout is not the one that decides the outcome, and health checks must cover the full node lifecycle.
IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU clustersame root cause · shared InfiniBand fabric, NCCL, PyTorch
IL-0025 — linked by the editor