IL-0050 Toyota Connected · 2018-11-05 (published)kubernetesnetworking
Spring Boot services on Kubernetes reached an Azure Redis Cache by a hostname with four dots. The pods used the Kubernetes default ndots:5, so the resolver tried the name with each cluster search domain before trying it as written. The services ran on Alpine images, whose musl resolver has documented DNS caveats when the first nameserver answer is NXDOMAIN, and used Netty 4.1.30, whose DNS resolver did not behave as expected (fixed in later versions). The author concludes that ndots was the catalyst and the combination of the three caused the repeated resolution failures.
Diagnosis took a full day and went through several wrong turns. The configuration was first pointed at a legitimate Redis Cache instance and dependent services were restarted, which seemed to fix it. When failures came back, a trailing dot was added to the Redis URI to force a fully qualified lookup; that helped in dev but collided with Netty, which rejected the trailing dot. The library's connection pool size was corrected in a new release, cutting the Redis connection count by about 70%.
The fix that restored most services was a dnsConfig block in the Kubernetes deployment manifests setting ndots to 4, so the four-dot Redis hostname is resolved as written. Other teams were told they might need the same setting; the team does not use in-cluster service discovery, so the risk of a lower ndots was acceptable. DNS resolution errors still occur from time to time, and the author expects moving from Alpine to slim Debian (glibc) images and to newer Netty versions to remove the rest.
Services that used the team's Redis-dependent library crashed on start-up, and their teams reported it to the ops members. The Spring Boot logs showed the applications failing to resolve the Redis hostname during bean initialisation.
A first fix seemed to work, but a few days later hostname resolution failures reappeared in Kibana. Adding a trailing dot to the Redis connection URI reduced the failures in the dev environment, yet the day after that change was merged the failures continued, and the problem became an incident. A Busybox pod started by hand resolved the Redis hostname without trouble.
The pods' /etc/resolv.conf read "nameserver 10.10.10.10", "search ns.svc.cluster.local svc.cluster.local cluster.local" and "options ndots:5". Deeper in the stack trace was the error "Server name value of host_name cannot have the trailing dot": the Netty version in use did not accept a hostname ending in a dot.
Two other problems turned up along the way. The Azure Redis Cache in the configuration turned out to be a previously unknown instance. The Redis connection count was close to 95% of its limit, because the library set the maximum connection pool size wrongly.
Symptom ? DNS resolution failuresIntermittent connection failures
Root cause ? Software defect
Service teams reported start-up crashes to the ops members of the team; later the resolution failures were seen in Kibana.
With ndots:5, a name outside the cluster that has fewer than five dots is first tried with every cluster search domain. Adding a trailing dot avoids that, but only if every library in the path accepts it, and the Netty version in use did not.
The ndots change is a workaround that will not hold forever. The author would like better alerting for this kind of failure but notes that alerts on one-off DNS failures are not wanted.
–