IL-0150 Amazon Web Services · 2025-10-19awskubernetesnetworking
DynamoDB keeps hundreds of thousands of DNS records for its load balancers in each region, maintained by automation in two parts. The DNS Planner watches load balancer health and capacity and produces new DNS plans; the DNS Enactor, running independently in three availability zones, applies each plan to Route 53 in a transaction. Before applying a plan, an Enactor checks once that it is newer than the plan already applied. That night one Enactor was badly delayed by retries while the Planner kept producing new plans and a second Enactor applied a much newer one. When the second Enactor finished, its clean-up deleted plans far older than its own. At the same moment the delayed Enactor, whose newer-than check was by now stale, wrote its old plan over the regional endpoint; the clean-up then deleted that plan, leaving the record for dynamodb.us-east-1.amazonaws.com empty. With the active plan gone, no Enactor could apply later plans, and the automation could not repair itself.
Engineers identified DynamoDB's DNS state as the source at 12:38 AM. By 1:15 AM temporary mitigations let some internal services connect and repaired internal tooling, and manual operator intervention restored all DNS information by 2:25 AM. Clients recovered as cached DNS records expired between 2:25 and 2:40 AM.
EC2 took much longer. The DropletWorkflow Manager (DWFM), which holds a lease on every physical server (droplet) that hosts instances, depends on DynamoDB for its periodic state checks, so leases across the fleet had slowly been timing out. When DynamoDB returned, DWFM tried to re-establish leases with so many droplets that the work timed out and was queued again, a congestive collapse with no established recovery procedure. At 4:14 AM engineers throttled incoming work and selectively restarted DWFM hosts, and by 5:28 AM all leases were back. The Network Manager then had a large backlog of network configuration for new instances, so new instances lacked connectivity until 10:36 AM. NLB's health checks brought those instances into service before their network state had propagated, which made health checks flap and triggered automatic availability zone DNS failover that removed capacity; engineers disabled automatic health check failover at 9:36 AM. Throttles were relaxed from 11:23 AM and EC2 was fully recovered at 1:50 PM.
ECS, EKS and Fargate in us-east-1 had container launch failures and cluster scaling delays from 11:45 PM PDT on 19 October until the services recovered at 2:20 PM PDT on 20 October.
At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.
EC2 API calls in us-east-1 returned more errors and took longer, and new instance launches failed with "insufficient capacity" errors, later also with "request limit exceeded" errors from throttling. Instances already running were not affected. From 5:28 AM new launches began to succeed, but new instances could lack network connectivity until about 10:36 AM.
From 5:30 AM some Network Load Balancers had more connection errors: health checks alternated between failing and passing, so NLB nodes and targets were removed from DNS and put back. Lambda, Amazon Connect, STS, IAM console sign-in, Redshift and AWS Support were also affected.
Between 11:45 PM PDT on 19 October and 2:20 PM PDT on 20 October, ECS, EKS and Fargate in us-east-1 had container launch failures and cluster scaling delays.
Symptom ? DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency
Root cause ? Software defect
DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.
A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result.
Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.
same root cause · shared Cloud load balancer
same root cause · shared Application database
same root cause · shared Cloud load balancer
same root cause · shared Application database
same root cause · shared Cloud load balancer