IL-0083 Adevinta · 2023-06-13 (published)kubernetesnetworking
SCHIP clusters resolve names through CoreDNS for cluster.local and Amazon's resolver for external names, with a dnsmasq agent on every node as a local cache. In the most affected cluster, which had the highest DNS query volume in the fleet because of its pod density, its applications and CronJobs that ran every few hours, three problems added up and caused intermittent 5xx errors for about a month.
First, dnsmasq allows 150 concurrent queries per node by default and the cluster exceeded it; the team raised the limit to 1500, as Google recommends. Second, dnsmasq cached only external answers: recent CoreDNS versions answer without the Recursion Available (RA) flag, and dnsmasq refuses to cache such answers, so every cluster.local query still went to CoreDNS. A CoreDNS header block that sets ra and aa and clears rd made dnsmasq cache local answers. Third, with the Kubernetes ndots setting, any name with fewer than five dots is first tried with every search domain, so short service names produced hundreds of unresolvable lookups per second.
The team made dnsmasq answer NXDOMAIN for those garbage names on the node instead of forwarding them, changed fluent-bit's log router address to a fully qualified name with five dots, and asked the affected tenants to use FQDNs. Requests reaching CoreDNS fell from more than 4000 per second to less than 1000 per second, and the DNS Latency SLI returned to above 99.9%. Two or three other large clusters had degraded SLIs from the same configuration.
On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.
The ingress controller was suspected first: its instance count had moved between two and five and showed a little CPU throttling. Raising its minimum instances and CPU removed the fluctuation and the throttling, but the errors continued. The errors then lined up with a recurring alert, "A fluent-bit agent stopped processing logs", from nodes where the affected service ran, and the team drained those nodes and restarted the Calico networking layer, as it had done for that alert before. That did not stop the reports either.
A dedicated investigation found that most of the errors came from one application, a proxy that called many microservices by bare Kubernetes service names such as http://foo/api. Fluent-bit also reached its log forwarder by a short service name. The cluster's DNS Latency SLI, the share of internal lookups answered within 16 ms and external lookups within 64 ms, had been low compared with other clusters for some time, and DNS misses rose whenever quality of service dropped.
The dnsmasq logs on the nodes showed "dnsmasq[25]: Maximum number of concurrent DNS queries reached (max: 150)". After that limit was raised, the SLI improved and then slid back over a week; after the caching fix it degraded again a week later. Running DNStop on problem nodes showed hundreds of requests per second for names that cannot resolve, such as cluster.local.cluster.local, schip.io.svc.cluster.local and grafana.net.platform-services.svc.cluster.local.
Symptom ? High error rateDNS resolution failuresHigh latency
Root cause ? DNS or network fault
A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.
–
The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated.
The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did.
With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.
same root cause · shared CoreDNS
same root cause · shared Client-side DNS resolver and cache