Incident Library?

API · MCP 한국어

Incident Library?

API · MCP 한국어

Component ?

Slurm job scheduler

Slurm 작업 스케줄러 · slurm

1 case · 2022

Root cause ?Hardware failure

Contributing factor ?Unreviewed defaults Limited debug access

Symptom ?GPU unavailable Training job failure

Cases, by severity

IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022
Root cause Hardware failure

View on the graph