IL-0078 ZenML · 2025-05-11kubernetesaws

Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces

The production EKS cluster that hosts ZenML Pro workspaces was on an outdated Kubernetes version and AMI type. On 9 May 2025 the staging cluster was fully upgraded with little disruption, which also showed that the EKS and VPC Terraform modules had to be upgraded to finish the cluster upgrade. The production upgrade was approved for the weekend of 10 to 11 May.

The upgrade started at 18:00 on 10 May while most workspaces were idle, and the VPC and EKS module upgrades went through cleanly. At 23:38 KMS secrets encryption was enabled through Terraform as part of the EKS module upgrade; it took much longer than on staging, likely because of the number of nodes and secrets. From 00:45 the upgrade was left to run overnight, since no problems were expected.

Applying the encryption through Terraform hit an undocumented bug: Terraform also removed key IAM permissions from the worker nodes, which made the workspace server pods unreachable from outside the cluster. Editing the IAM roles did not restore access; the node group had to be deleted and recreated with the correct IAM permissions and configuration, and service was fully restored around 07:30. The report counts about eight hours of downtime for all production workspaces, with no data loss, corruption or unauthorised access.

The report names three contributing factors: outdated EKS and VPC Terraform modules and legacy AMI versions, upgrade complexity that built up because Kubernetes and module versions had been skipped, and a risk judgement based on a staging cluster of much smaller scale.

Severity: –
Scope: all production ZenML Pro workspaces (workspace UIs and pipelines); the ZenML Pro dashboard was unaffected
Users affected: all ZenML Pro workspace customers
Region: –
Time to detect: 6h
Time to resolve: 8h

Symptoms

Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.

When engineers resumed an overnight cluster upgrade at 06:00, they found the production EKS cluster degraded. The step that enabled secrets encryption with AWS KMS had succeeded, but Terraform had also removed key IAM permissions from the EKS worker nodes, and the ZenML Pro server pods could no longer be reached from outside the cluster.

Changing the IAM roles to restore access did not work. Between 06:00 and 07:30 the outage continued while the team troubleshot it.

Symptom ? Service outage

Root cause

Root cause ? Software defect

Component ?Terraform Node pool

Contributing factor ?Unrepresentative staging Deferred upgrades

Trigger: At 23:38 on 10 May, secrets encryption with AWS KMS was enabled through Terraform as part of the EKS module upgrade.

Detection

Engineers found the cluster degraded when they resumed the upgrade at 06:00.

Remediation

deleted and recreated the EKS node group with the correct IAM permissions and configuration
restored service access across all affected ZenML Pro environments

Action items

Upgrade all Terraform modules (EKS, VPC) to current supported versions done
Complete the EKS cluster upgrades as planned done
Review the strategy for periodic Kubernetes and infrastructure upgrades to prevent a recurrence planned

Lessons

A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.

Sources

https://status.zenml.io/incidents/bnrlb8tck3v0 · official-status

Related cases

IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025
same root cause · shared Node pool
IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024
same root cause · shared Node pool
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022
same root cause · shared Terraform
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020
same root cause · shared Node pool