Incident Library
?
API · MCP
한국어
Incident Library
?
API · MCP
한국어
Root cause
?
Dependency failure
의존성 장애 ·
dependency-failure
3 cases · 2024 to 2026
Component
?
GPU driver
2
InfiniBand fabric
2
NCCL
2
PyTorch
2
Node operating system image
1
Node pool
1
Contributing factor
?
Monitoring blind spot
2
Unverified assumption
1
Automation amplification
1
Unreviewed defaults
1
Partial rollout
1
Symptom
?
GPU unavailable
2
Link flapping
2
NCCL timeout
2
Node NotReady
1
Throughput degradation
1
Training job failure
1
Cases
IL-0023
A training run failed on an NCCL timeout from one brief InfiniBand port drop
— Databricks, 2026, sev4
IL-0025
Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster
— Imbue, 2024
IL-0021
GKE GPU nodes came up without a working NVIDIA driver
— Google Cloud, 2024, sev2
View on the graph