IL-0171 Inflab · 2025-08-27 (published)kubernetesawsai-infra
Inflab runs AI upscaling of course videos on Amazon EKS 1.32 with g5g.xlarge GPU instances, the Bottlerocket AMI amazon/bottlerocket-aws-k8s-1.32-nvidia-aarch64-v1.39.1-3a880b44, and Karpenter v1.4.0 managing nodes. The team had turned on Karpenter's node auto-repair, an alpha feature since Karpenter v1.1.0, to deal with "zombie" nodes: when kubelet was OOM-killed on Bottlerocket, the EC2 instance stayed up but the node went to the Unknown state.
The Bottlerocket NVIDIA AMI in use did not include libdcgm.so, so DCGM failed to initialise and the node's AcceleratedHardwareReady condition turned False about a minute after the node was Ready. Node auto-repair counts AcceleratedHardwareReady=False as unhealthy, alongside Ready=False or Unknown and the memory, disk, PID and network pressure conditions. The repair controller waits for a toleration period, about 10 minutes here, after the condition's last transition and then deletes the NodeClaim, so each GPU node was terminated and replaced by an identical node with the same problem, in an endless loop.
The team considered installing the DCGM library by hand, which Bottlerocket's immutable design makes hard, and switching to another AMI, which would mean large infrastructure changes. It chose to disable node auto-repair by setting nodeRepair: false in the Karpenter settings, accepting that this is a temporary measure. The original zombie-node problem was mitigated separately by using instance types with at least 4 GB of memory.
GPU nodes for an AI video upscaling job were created normally and reached Ready, and then about 10 to 11 minutes later Karpenter treated each one as unhealthy and replaced it. Every new node went the same way, so the GPU pods kept restarting and the upscaling work could not make progress.
The node events showed the sequence. kube-proxy started, kubelet posted "kubelet is posting ready status" and the node became Ready. About a minute later an AcceleratedHardwareReady event recorded by Karpenter showed the condition changing from True to False with Reason "DCGMError" and Message "failed to initialize DCGM: libdcgm.so not Found". Karpenter then logged NodeRepairBlocked with "more than 20% nodes are unhealthy in the nodepool", and about ten minutes after that FailedDraining ("Failed to drain node, 1 pods are waiting to be evicted") and InstanceTerminating ("Instance is terminating").
The missing library is used by NVIDIA DCGM (Data Center GPU Manager) to collect GPU health and events. Its absence does not stop GPU computation itself, but it makes DCGM-based GPU health monitoring impossible, and the node condition reported the accelerator as not ready.
Symptom ? Node recreation loopGPU unavailable
Root cause ? Software defect
The team noticed GPU nodes being replaced repeatedly after running GPU pods, then found the DCGMError message in the node events.
–
Automatic node repair acts on every unhealthy node condition, including AcceleratedHardwareReady, so an image defect that only breaks GPU health monitoring is enough to have every GPU node terminated and recreated in a loop. Before turning on auto-repair, check which conditions it watches and whether the node image reports them correctly.
Disabling auto-repair stops the loop at once but is only a temporary measure; installing the missing library or changing the AMI address the cause but cost more on an immutable OS such as Bottlerocket.
same root cause · shared Node operating system image
same root cause · shared GPU driver
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node operating system image
same root cause · shared Node operating system image