Incident Library?

Incident Library?

Contributing factor ?

Unverified assumption 미검증 가정 · `assumption-unverified`

2 cases · 2019 to 2024

Root cause ?Dependency failure 1 Software defect 1

Component ?Configuration templating 1 GPU driver 1 HAProxy 1 InfiniBand fabric 1 Ingress controller 1 NCCL 1 PyTorch 1

Symptom ?GPU unavailable 1 High latency 1 Intermittent connection failures 1 Link flapping 1 NCCL timeout 1 OOMKilled 1 Throughput degradation 1 Load imbalance 1

Cases

IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster — Imbue, 2024
Root cause Dependency failure
IL-0011 Sorted HAProxy backends sent every request to the same first pod — Skyscanner, 2019, sev2
Root cause Software defect

View on the graph