IL-0108 DFDS · 2020-11-04kubernetes
DFDS held an Alcide workshop and used the production cluster Hellman as the sandbox. At 13:13 CET an Alcide admission controller was applied, on the understanding that it would not be disruptive; the team felt safe because it was re-applying something that had been there before. The admission controller was suspected of responding slowly, and every create and update request it covered timed out.
Users noticed when their deployments timed out, 88 minutes after the change. After confirming the problem and toggling the admission controller to test the fix, the team deleted it for good at 15:39. The postmortem also notes that no smoke tests existed to catch broken capabilities, and that knowledge of Alcide had not been spread across the team because the tool was about to be deprecated. Time to detection was 88 minutes and time to recovery 122 minutes.
At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.
Create and update operations timed out, mostly on Deployments but also on other resources in the API groups "", "apps", "batch" and "extensions". The team suspected the Alcide admission controller applied earlier that day of answering too slowly, and at 15:15 switched it off and on to check.
Symptom ? Deploys failingControl plane lag
Root cause ? Configuration change
Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.
New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.
same root cause · shared Admission webhook, Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server