Incident Library?

API · MCP 한국어

Incident Library?

API · MCP 한국어

Root cause ?

Hardware failure

하드웨어 고장 · hardware-failure

3 cases · 2022 to 2024

Component ?NCCL 2 PyTorch 2 ConfigMaps and Secrets 1 etcd 1 GPU hardware and memory 1 Kubespray 1 Node root disk 1 Slurm job scheduler 1

Contributing factor ?Single point of failure 2 Unreviewed defaults 1 Limited debug access 1

Symptom ?GPU unavailable 2 Training job failure 2 NCCL timeout 1 Node NotReady 1

Cases

IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024
IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joinedHans H. Kim · 2023
IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

View on the graph