IL-0105 DFDS · 2021-06-22kubernetesawsnetworking
DFDS was refactoring and cleaning up its infrastructure code so that EKS add-ons would be managed by native Terraform instead of kubectl. As part of that work the pipeline replaced the VPC CNI at 08:21 UTC. The change to the VPC CNI detached the target groups from the Application Load Balancer, so all inbound traffic failed to reach the cluster and every service answered 503.
The postmortem names two contributing factors: the de-registration delay of the ALB target groups had earlier been lowered permanently to speed up node roll-overs, and an essential network component was replaced. The team committed a fix at 08:33 that reverted the infrastructure-modules reference, re-ran the pipeline after an AWS API timeout, and at 08:46 the pipeline reattached the target groups to the ALB. Time to detection was 8 minutes and time to recovery 25 minutes.
At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.
Eight minutes before the alerts, at 08:21, the deployment pipeline had replaced the VPC CNI. The investigation started at 08:30, and when the fix was pushed through the pipeline it first failed on an AWS API timeout and had to be started again at 08:38.
Symptom ? Service outageHigh error rate
Root cause ? Configuration change
Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.
The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.
same root cause · shared Cloud load balancer
same root cause · shared AWS VPC CNI
same root cause · shared AWS VPC CNI
same root cause · shared AWS VPC CNI
same root cause · shared Terraform
same root cause · shared Terraform
same root cause · shared Cloud load balancer
IL-0208same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Terraform
same root cause · shared Cloud load balancer