IL-0049 Preply · 2020-02-28kubernetesnetworking
At 22:13 EET on 28 February 2020 the CoreDNS autoscaler at Preply scaled CoreDNS from three pods to two. kube-proxy tried to delete the conntrack entries for UDP connections to the removed pod and failed, so the kernel's connection tracking table kept sending some services' DNS queries to an address where no CoreDNS pod was running. conntrack is the Linux kernel table that remembers the address translation chosen for a flow so later packets skip the lookup. Those services could not resolve names, returned 5xx errors, and 15000 events were dropped across services A, B and C.
On-call engineers were paged at 22:18, started troubleshooting at 22:21, and reverted the latest release of the service at 22:39; the errors were gone at 22:40. The postmortem gives the resolution as a regular deploy that created new nodes, after which the CoreDNS autoscaler added pods and the conntrack table was rewritten. The team lists time to detect as 4 minutes, time to engage as 21 minutes, time to fix as 1 minute, and the total as 26 minutes.
The team did not find the real root cause and suspected a specific conntrack bug. The postmortem also admits that DNS problems were known beforehand but not prioritised, and that every action item treats consequences rather than the root cause.
Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.
DNS lookups from some services inside the Kubernetes cluster were failing because their traffic to the cluster DNS was still being sent to a CoreDNS pod that no longer existed. kube-proxy had logged the failure at the moment of the scale-down: "Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...". The CoreDNS deployment events showed "Scaled down replica set coredns-6cbb6646c9 to 2" a fraction of a second earlier.
Not every service was affected, and no node reached any of its limits.
Symptom ? DNS resolution failuresHigh error rateStale endpoint routing
Root cause ? DNS or network fault
Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.
Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.
same root cause · shared kube-proxy, conntrack table
same root cause · shared CoreDNS
same root cause · shared conntrack table
same root cause · shared conntrack table