IL-0109 DFDS · 2023-02-14kubernetesaws
A scheduled node rollover on 13 February 2023 set containerd as the default container runtime by mistake, and as a result Kubernetes logs stopped being shipped to CloudWatch. To fix the logs, the team started an unscheduled node rollover the next morning that swapped the runtime back. The value used in the eks-bootstrap script was believed to be docker-shim but needed to be dockerd, so the new nodes never joined the cluster and the rollover timed out with them still in the Auto Scaling group.
The team then ran the EKS pipeline without a rollover to apply a launch template with dockerd. Terraform saw the desired size of the Auto Scaling group still at the rollover value of 30 and set it back to 15. The group's default termination policy removes the instances that use the oldest launch template first, and the oldest ones were the healthy nodes, not the faulty new ones that had not been removed yet. All healthy worker nodes were terminated and the cluster was left with none.
A first run of the node rollover script at 10:49 failed on a timeout from Docker Hub; a second run at 10:52 added new nodes from 10:57, and at 11:01 the load balancer marked them healthy and traffic flowed again. Recovery took 13 minutes. A Sev1 ticket was raised at 11:07 and closed at 11:59.
At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.
An unscheduled node rollover to switch the runtime back started at 10:29 and failed on a timeout at 10:39. The runtime value used was wrong, so the new nodes never joined the cluster and were left as "zombies".
At 10:45 the EKS pipeline was run again without a node rollover, to apply the new launch template with dockerd as the runtime. Because the failed rollover had never set the Auto Scaling group's desired size back, Terraform changed it from 30 to 15. At 10:46 all the healthy nodes were terminated, since the group's default termination policy removes the instances with the oldest launch configuration or template first. By 10:48 the cluster had no worker nodes.
Root cause ? Configuration change
End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.
When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way.
The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.
same root cause · shared Telemetry agent, Container runtime
same root cause · shared Node launch template, Node pool
same root cause · shared Node pool, Terraform
same root cause · shared Telemetry agent
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Terraform
same root cause · shared Terraform
same root cause · shared Node pool