DNS or network fault

Cases

1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycledDigitalOcean · 2024
2. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024
3. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023
4. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022
5. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020
6. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the nodeMindTickle · 2020
7. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020
8. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019
9. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019