IL-0111 DFDS · 2021-08-26kubernetesawsnetworking
On August 26, 2021, as part of moving off Traefik v1, DFDS made Traefik v2 the main Kubernetes ingress controller by pointing the AWS load balancers at it. A misconfiguration kept Traefik v2 from forwarding the X-Forwarded headers, so x-forwarded-proto reached applications as http instead of https. The Okta authentication workflow broke for every service that used Okta, and the online booking system was completely down for a little over an hour.
Functional tests after the switch looked fine, and the problem was noticed 64 minutes later through a developer's report of many HTTP 307 redirects. Because the booking system was critical, the team rolled back before investigating further, and service was back at 09:49 UTC, 87 minutes after the change. The root cause was found afterwards in a deep dive with the Okta team. Among the contributing factors, DFDS lists the overlooked header change, the long detection time, the expectation that every team watches its own application after critical changes, and Slack announcements that get lost among other messages.
After the switch to Traefik v2, functional tests indicated that everything was fine, and this was reported on Slack at 08:44 UTC. The first sign of trouble came at 09:26 UTC, when a developer reported a high count of HTTP 307 redirects, especially on the Okta service. At 09:32 UTC the team learned that the online booking system was down.
The cause turned out to be request headers: Traefik v2 was configured so that it did not forward the insecure "X-FORWARDED-" headers, and the x-forwarded-proto header that applications received changed from https to http. That broke the Okta authentication workflow for every service using Okta. The headers had been checked before the migration, but this change was missed.
Symptom ? Service outage
Root cause ? Configuration change
A developer reported a high count of HTTP 307 redirects, in particular on the Okta service, at 09:26 UTC, 64 minutes after the change; six minutes later the team was told the online booking system was down. The functional tests run after the switch had not caught it.
Cloud Engineering cannot test every application scenario on its own, even after testing its own applications for weeks. A header that changes value behind a new ingress controller, here x-forwarded-proto going from https to http, can break an authentication flow while basic functional tests still pass. DFDS asks whether diffing request headers before and after the migration would have shown the problem in advance.
same root cause · shared Cloud load balancer, Ingress controller
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer