Incident Library?

API · MCP 한국어

Incident Library?

API · MCP 한국어

Component ?

GPU hardware and memory

GPU 하드웨어·메모리 · gpu-hardware

2 cases · 2021 to 2024

Root cause ?Dependency failure 1 Hardware failure 1

Contributing factor ?Limited debug access 1 No runbook 1 No headroom 1 Single point of failure 1

Symptom ?GPU unavailable 2 Training job failure 2 NCCL timeout 1

Cases, by severity

IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024
Root cause Hardware failure
IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021
Root cause Dependency failure

View on the graph