IL-0182 이언철 · 2024-04-16 (published)kubernetesaws
The author, a DevOps engineer, describes CoreDNS problems on an Amazon EKS cluster that uses Karpenter. Under fast-changing traffic, too few CoreDNS pods handled the DNS load: latency rose, the network dropped intermittently for task pods, and the horizontal pod autoscaler's scale-out drove a surge in database connections that spread the impact across the service.
The first round of fixes sized CoreDNS to its capacity and made it scale with the cluster. The author notes that each CoreDNS pod needs a certain amount of resources and that average latency and queries per second depend on how much each pod can handle. The team set up the Cluster Proportional Autoscaler (CPA), which scales CoreDNS by cluster size, choosing nodesPerReplica and coresPerReplica for the instance types in use and tuning the replica minimum and maximum through monitoring. They spread CoreDNS with topologySpreadConstraints instead of anti-affinity, as Karpenter suggests, using the soft ScheduleAnyway setting so Karpenter could still consolidate and replace nodes for drift; EKS add-on versions v1.9.3-eksbuild.11 and v1.10.1-eksbuild.7 and later include topologySpreadConstraints by default. They also set alert thresholds on CoreDNS error rate, panics, request duration and request rate, and in New Relic used a Prometheus agent to integrate kube-dns with an agent status dashboard.
The over-provisioned CoreDNS pods then piled up 3 to 4 per node on the few nodes they could run on, and those nodes could not be consolidated. To take load off the main CoreDNS, the team deployed NodeLocal DNSCache, a CoreDNS DaemonSet on every node that answers from its cache or forwards to the main CoreDNS. DNS query volume to the main CoreDNS fell sharply, spikes became easier to absorb, and a DNS problem now stays on one node instead of spreading to the whole service. After this change the production metrics improved considerably and the node over-expansion stopped. The author concludes that CPA together with NodeLocal DNSCache resolved the CoreDNS load problem.
When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.
Task pods, which start when a request comes in, ran on nodes that come and go. The small number of CoreDNS pods could not keep up with them, so the network dropped intermittently and these pods did not work properly.
After the first fixes, a second problem appeared. CoreDNS replicas had been raised well beyond need for stability. Every node carried a taint, so the CoreDNS Deployment's pods landed only on particular nodes (for example the default ones), and to keep topologySpreadConstraints maxSkew: 1 the scheduler placed 3 to 4 CoreDNS pods per node. Those nodes could not be consolidated, and the node count grew too large.
Symptom ? High latencyIntermittent connection failuresAutoscaler misbehaviour
Root cause ? Capacity exhaustion
–
–
A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.
same root cause · shared Horizontal pod autoscaler, CoreDNS
same root cause · shared CoreDNS
same root cause · shared Horizontal pod autoscaler
same root cause · shared Pod scheduling and placement
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared Karpenter
same root cause · shared Pod scheduling and placement
same root cause · shared CoreDNS
same root cause · shared Pod scheduling and placement