IL-0112 DFDS · 2022-10-31kubernetesaws
DFDS's Terraform infrastructure-modules were hard-coded to use the latest AWS EKS node AMI as soon as it was available, with no way to pin a specific one, and a daily rollover replaced the cluster's nodes. On 2022-10-31 the rollover picked up AMI v20221027, which was later recalled because of a kernel issue. Nodes on that image had sporadic communication problems: they briefly reported "PLEG is not healthy", sometimes went unhealthy, the number of sockets per node kept growing, and one team saw network timeouts from its pods to other services.
The same day the team upgraded the vpc-cni add-on, the upgrade stalled, and the node flapping was first blamed on it. The team completed the upgrade by cordoning, draining and deleting the stuck nodes one at a time, 15 in total, by 17:35. At 17:21 a GitHub issue about PLEG errors with this AMI led the team to learn that the AMI had been recalled. With no service loss visible, the team decided to monitor and wait for a new AMI release. As socket use kept rising and Kubernetes events reported communication problems, the team decided to roll back on 2022-11-02 at 11:56. A Terraform change that allowed setting a specific AMI was merged at 13:14, the pipeline rolled the nodes back to the previous AMI at 13:46, and at 14:40 the team judged the problem resolved. Time to detection was 7 hours and time to recovery 54 hours 10 minutes.
The write-up lists more factors that made the incident worse or harder to handle: some team applications had no readiness or liveness probes; it happened during an IT conference that most development teams and members of the reporting team itself attended; there were no on-call escalation paths with individual teams; the team that reported problems was hard to get specifics from; upgrading an EKS component on the same day as the new AMI confused the diagnosis; alerting did not catch the NodeNotReady events or the socket growth; and the team lacked deeper knowledge of Kubernetes components.
On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.
From 14:30 some nodes reported NodeNotReady for about 30 seconds at a time and then NodeReady again, with the message "KubeletNotReady PLEG is not healthy". The team assumed this came from the vpc-cni upgrade in progress. Some aws-node pods made progress and then got stuck again, and a terminating pod that was force-deleted was replaced by a new one that never became Ready. Logs for some containers could not be read through the Kubernetes API, although the aws-node containers were running and logging on the nodes themselves.
After the upgrade was finished by hand the NodeNotReady events continued, but no loss of service was seen or reported. On 2022-11-01 at 11:12 the team saw the number of sockets used per node rising continuously. On 2022-11-02 at 10:52 one team reported connectivity problems from some of its pods, network timeouts when talking to other services, but gave no details to investigate.
Symptom ? Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures
Root cause ? Software defect
The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.
When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters.
Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.
same root cause · shared Linux kernel, Node operating system image
same root cause · shared Linux kernel, kubelet
same root cause · shared Linux kernel, Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Terraform
same root cause · shared Linux kernel
same root cause · shared kubelet
same root cause · shared Node operating system image
same root cause · shared kubelet
same root cause · shared kubelet