Incident Library?

Incident Library?

Contributing factor ?

Partial rollout 부분 배포 · `partial-rollout`

1 case · 2024

Root cause ?Dependency failure

Component ?GPU driver InfiniBand fabric NCCL PyTorch

Symptom ?GPU unavailable Link flapping NCCL timeout Throughput degradation

Cases

IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster — Imbue, 2024
Root cause Dependency failure

View on the graph