Terraform

Cases, by severity

1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

   Root cause Configuration change

2. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

   Root cause Operator error

3. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025

   Root cause Software defect

4. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

   Root cause Configuration change

5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

   Root cause Software defect

6. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

   Root cause Configuration change

7. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one