IL-0080 Mercari · 2025-06-04 (published)kubernetesgcpnetworking
Mercari runs its services on managed GKE clusters, where kube-dns comes pre-installed in the kube-system namespace and is scaled by the default kube-dns autoscaling. Every pod on every node sent its DNS queries for names such as [service name].[namespace].svc.cluster.local to a small number of kube-dns pods. At peak traffic those pods were overwhelmed, which produced NXDOMAIN errors, truncated responses and "failed to refresh DNS cache" errors, and finally a Sev1 incident in which services failed DNS resolution and the failures cascaded into a production outage.
The Platform Engineering team responded by deploying NodeLocal DNSCache, which runs a DNS cache on each Kubernetes node so that pods on that node ask the local cache before kube-dns. The team first deployed it on the Laboratory GKE cluster, which only platform teams use for internal testing, with a specific nodeAffinity, and then rolled it out gradually to the Dev and Prod environments by adding labels to node pools so that the NodeLocal DNSCache pods could be scheduled there.
After the rollout, DNS calls to kube-dns fell by 10x and the DNS query rate of services on the cluster fell by 93%. DNS-level errors fell by 10x to 100x depending on the error class (10x for NXDOMAIN, 100x for truncated responses), and the "failed to refresh DNS cache" errors disappeared completely.
During peak hours, when Mercari's GKE clusters handle very high request rates, the platform team saw kube-dns reaching its limits. DNS query rates were high enough to overwhelm the kube-dns service.
The errors showed up at the DNS level: frequent NXDOMAIN answers, truncated responses, and a recurring "failed to refresh DNS cache" error that led to cache misses. The author says this last error was behind many production incidents.
The problem peaked in a Sev1 incident. Several services began to fail DNS resolution, their calls timed out, and because the microservices depend on each other the failures cascaded into a production outage.
Symptom ? DNS resolution failuresIntermittent timeoutsService outage
Root cause ? Capacity exhaustion
During peak hours the team saw frequent NXDOMAIN and truncated-response errors and recurring "failed to refresh DNS cache" errors on the clusters, which showed that kube-dns had reached its limits.
–
When every pod in a busy cluster resolves names through a few central kube-dns pods, DNS becomes a shared bottleneck, and a DNS failure quickly turns into higher latency, timeouts and cascading outages across microservices. A cache on each node takes most of that load off kube-dns and keeps resolution local to the node.
Rolling the cache out first to an internal test cluster with nodeAffinity, and then node pool by node pool through labels, kept the blast radius of the change small.
same root cause · shared kube-dns