IL-0102 Buildkite · 2026-08-25kubernetesaws
Buildkite had spent four months moving its production workloads from AWS ECS to an AWS EKS cluster, and the cluster had grown steadily throughout. Services in it locate databases, queues and other components through CoreDNS, which ran as a fixed set of three pods and did not scale with the size or rate of change of the cluster. Buildkite had also recently moved time-sensitive notification jobs from the general background worker pool into a new low-latency pool, and had given both pools the same high maximum replica count as the original shared pool, planning to lower both later.
At 22:44 UTC an application deploy consumed the headroom on the deployed nodes. Background workers, including the notification workers, were trying to scale out at the same moment. Their new compute was delayed while the cluster scaled, and since the metrics that had made them scale had not changed, they requested still more pods. The services' maximums would normally have capped this runaway scaling, but they had been set higher than usual. The flood of new pods, endpoints and nodes drove CoreDNS query time from under 1 ms to about 780 ms; queued queries filled memory, all three pods exceeded their memory limits and were restarted, and retries kept them from recovering. With DNS failing, APIs, job dispatch and notifications failed for all customers from 22:47 to 23:16 UTC.
The post-incident analysis found that a defect in a key monitoring query had hidden an upward trend in CoreDNS query duration that tracked the growing cluster size, and that DNS query volume, which had been rising as more workloads moved to EKS, was not monitored well enough. On the client side, recent application performance gains offset the slower DNS answers, so the trend drew no attention, and the planned work to autoscale CoreDNS had not been given priority. Engineers paused deploys, added CoreDNS replicas with more memory and enlarged the node pool that runs them. The new CoreDNS pods were serving by 23:12 UTC, DNS errors dropped quickly, and the accumulated backlog was processed by 23:16 UTC.
From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.
Webhooks from source control providers kept arriving, but their processing fell behind. A subset of customers whose database queries went through PgBouncer instances inside the EKS cluster got error responses from the web UI and on webhook delivery; these errors were less than 1.5% of all requests during the outage. A status post at 23:38 UTC put the failure down to Buildkite's internal DNS services.
Inside the cluster, CoreDNS kept completing queries at a steady rate, but the time to process each query rose from under 1 ms to about 780 ms. Pending queries piled up in memory until all three CoreDNS pods went over their memory limits and Kubernetes restarted them. Continued demand and DNS retries kept CoreDNS from recovering after the restarts.
Symptom ? Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour
Root cause ? Capacity exhaustion
The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.
A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart.
Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.
same root cause · shared Container memory limit, CoreDNS, Node pool
same root cause · shared CoreDNS, Node pool
same root cause · shared CoreDNS, Node pool
same root cause · shared Container memory limit, Node pool
same root cause · shared Horizontal pod autoscaler, CoreDNS
same root cause · shared Container memory limit, Node pool
same root cause · shared Container memory limit, Node pool
same root cause · shared Container memory limit, CoreDNS
same root cause · shared Horizontal pod autoscaler
same root cause · shared Node pool
same root cause · shared CoreDNS
same root cause · shared CoreDNS
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Container memory limit
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Container memory limit