Incident Library?

Incident Library?

Symptom ?

GPU unavailable GPU 인식 불가 · `gpu-unavailable`

2 cases · 2024

Root cause ?Dependency failure 2

Component ?GPU driver 2 InfiniBand fabric 1 NCCL 1 Node operating system image 1 Node pool 1 PyTorch 1

Contributing factor ?Unverified assumption 1 Automation amplification 1 Monitoring blind spot 1 Partial rollout 1

Cases

IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster — Imbue, 2024
Root cause Dependency failure
IL-0021 GKE GPU nodes came up without a working NVIDIA driver — Google Cloud, 2024, sev2
Root cause Dependency failure

View on the graph