Incident Library?

Incident Library?

Component ?

GPU driver GPU 드라이버 · `gpu-driver`

2 cases · 2024

Root cause ?Dependency failure 2

Contributing factor ?Unverified assumption 1 Automation amplification 1 Monitoring blind spot 1 Partial rollout 1

Symptom ?GPU unavailable 2 Link flapping 1 NCCL timeout 1 Node NotReady 1 Throughput degradation 1

Cases, by severity

IL-0021 GKE GPU nodes came up without a working NVIDIA driver — Google Cloud, 2024, sev2
Root cause Dependency failure
IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster — Imbue, 2024
Root cause Dependency failure

View on the graph