IL-0151 Amazon Web Services · 2023-06-13awskubernetes
AWS Lambda is built from cells. In each cell the Lambda Frontend receives and routes invocations, and the Lambda Invocation Manager manages the compute capacity, called execution environments, that runs customer code. From 10:01 AM PDT on 13 June 2023 the Frontend fleet in us-east-1 scaled out for normal daily traffic. At 11:49 AM one cell grew past a capacity threshold no single cell had reached before, which set off a latent software defect: execution environments were allocated for incoming requests but never fully used by the Frontend. Lambda could not provision new execution environments, so invocations in that cell failed or slowed down. Invocations in other cells were not affected.
Engineers identified the defect by 12:26 PM. Having confirmed that traffic had subsided, they scaled the Frontend fleet down to a size that no longer triggered it. New invocations began to recover at 1:30 PM and synchronous invocations were fully recovered at 1:45 PM, when most dependent services, including EKS cluster provisioning, recovered too. Lambda then worked through the backlog of asynchronous events according to each event source's retry policy, and by 3:37 PM Lambda and all dependent services were operating normally.
AWS disabled the Frontend scaling activity that set off the event, fixed the latent bug and deployed the fix to all Regions. The event also exposed a gap in Lambda's cellular architecture for scaling the Frontend, which let a latent bug cause impact as a cell grew.
From 11:49 AM PDT on 13 June 2023, Lambda function invocations in the Northern Virginia (us-east-1) Region returned more errors and took longer. Customers that triggered functions from asynchronous or streaming event sources saw their event backlogs grow because the events were not being processed.
Services that depend on Lambda were affected too. Amazon EKS had higher error rates and latency when provisioning new clusters, while existing EKS clusters were not affected. Amazon STS had elevated error rates in three separate periods between 11:49 AM and 2:10 PM, and SAML federation through AWS Sign-in failed more often between 1:13 PM and 2:11 PM, when throttles were put on STS; existing IAM sessions kept working. EventBridge deliveries to Lambda were delayed by up to 801 seconds.
The AWS Management Console for us-east-1 showed either 'AWS Management Console is currently unavailable' or a '504 Time-out' error page between 11:48 AM and 2:02 PM. Amazon Connect calls failed to connect and chats and tasks failed to start, and AWS Support Center call and chat were unavailable until dependent services recovered.
Symptom ? High error rateHigh latencyCredentials unavailable
Root cause ? Software defect
Engineering teams were engaged immediately and began investigating; by 12:26 PM PDT they had identified the latent defect and its effect on provisioning compute capacity.
A fleet that scales automatically can grow into a size that was never tested, and a defect that only appears at that size stays hidden until ordinary daily growth reaches it. AWS's answer is to cap every cell at a well-tested size.
For EKS users, the event shows that provisioning new clusters in us-east-1 depended on Lambda, while clusters that were already running were not affected.
–