IL-0106 DFDS · 2021-09-06kubernetesawsnetworking
DFDS routed ingress traffic for its Kubernetes services through Traefik v2 behind an AWS ALB. On 2021-09-01 a malformed IngressRoute manifest was applied, and the official Traefik v2 API schema did not stop it. Traefik v2 versions before 2.5 have a bug (Traefik issue 8098) that makes such a resource stop Traefik from receiving or routing any ingress traffic, but the bug only takes effect once all Traefik v2 pods have restarted. That happened during the automated node rollover on 2021-09-06, and from 08:00 UTC every Kubernetes service returned 404.
Because Traefik v2 failed gracefully, the team first chased the unhealthy ALB targets and changed the health check path, which did not bring traffic back. At 08:50 the team began rolling back to Traefik v1 as the main ingress controller, carried out the rollback by hand at 09:00, and ingress traffic worked again at 09:08. At 09:10 the team found the root cause by deleting the malformed IngressRoute by hand and confirmed the Traefik v2 bug, and at 09:20 the changes were applied upstream. Detection took 5 minutes and recovery 68 minutes.
At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.
At 08:10 the targets of the AWS Application Load Balancer (ALB) in front of the cluster were unhealthy. Changing the ALB health check path made the targets healthy again at 08:15, but at 08:20 all services were still returning 404. Traefik v2, the ingress controller, failed gracefully, which pointed the investigation at the ALB, and other checks up to 08:30 found nothing conclusive.
The resource behind the failure had been applied five days earlier: a malformed IngressRoute went into the cluster at 11:34 UTC on 2021-09-01. The failure appeared only after the automated node rollover that started at 07:30 on 2021-09-06 had restarted all the Traefik v2 pods.
Root cause ? Software defect
All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.
A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik.
On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.
same root cause · shared Cloud load balancer
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Cloud load balancer
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Cloud load balancer
same root cause · shared Ingress controller