Incident Library?

Incident Library?

Contributing factor ?

Monitoring blind spot 모니터링 사각지대 · `monitoring-blind-spot`

7 cases · 2017 to 2026

Root cause ?Configuration change 2 Dependency failure 2 Software defect 2 Capacity exhaustion 1

Component ?InfiniBand fabric 2 NCCL 2 PyTorch 2 Argo CD 1 CoreDNS 1 Container CPU limit 1 Client-side DNS resolver and cache 1 etcd 1 GPU driver 1 Kubernetes API server 1 kubelet 1 Linkerd 1 Linux kernel 1 Container memory limit 1 Model request routing 1 Persistent volumes and stateful sets 1 TPU 1 XLA compiler 1

Symptom ?High error rate 4 Link flapping 2 NCCL timeout 2 Service outage 2 CPU throttling 1 DNS resolution failures 1 GPU unavailable 1 High latency 1 Intermittent timeouts 1 Monitoring outage 1 OOMKilled 1 Probe failures 1 Misrouted requests 1 Resources deleted 1 Response quality degradation 1 Stale endpoint routing 1 Throughput degradation 1 Training job failure 1

Cases

IL-0023 A training run failed on an NCCL timeout from one brief InfiniBand port drop — Databricks, 2026, sev4
Root cause Dependency failure
IL-0022 Three overlapping routing and TPU bugs degraded model responses — Anthropic, 2025, sev3
Root cause Configuration change
IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster — Imbue, 2024
Root cause Dependency failure
IL-0009 A generated name collision made Argo CD delete the production environment — InfluxData, 2023, sev1
Root cause Configuration change
IL-0012 CPU limits and a kernel scheduler bug stalled multi-threaded services — Omio, 2020, sev3
Root cause Software defect
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019
Root cause Capacity exhaustion
IL-0003 Payment failures that grew into a full platform outage — Monzo, 2017, sev1
Root cause Software defect

View on the graph