IL-0138 Barush Mendez · 2025-03-03kubernetes
The single master node of a homelab Kubernetes cluster has an Nvidia GPU for hardware acceleration in Jellyfin. Weeks before the incident, installing the nvidia-dkms-390 driver failed because it would not build against the installed kernel, and the kernel headers and other dependencies were left unconfigured. The failure was not dealt with, and the node kept running normally until a power outage on 2025-03-03 rebooted it. On boot the kernel panicked and the node could not start.
Because it was the only master node, the Kubernetes API was gone and every hosted application became unreachable, as services relied on the master node's private IP. There were no automated alerts, so the outage was found manually after 3 hours, and recovery took until 2025-03-09, six days in total. Once the kernel panic was seen on a monitor, booting an older kernel and repairing the broken packages fixed the node. No data was lost.
Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.
The master node could not be reached remotely. With a monitor attached, its boot logs showed a kernel panic during boot. The node had been restarted by a power outage, and it was the only master node; the hosted services also depended on its private IP address, so its failure took down everything.
Symptom ? Control plane unresponsiveService outageMonitoring outage
Root cause ? Software defect
Manually, when someone tried to access the services; there were no automated alerts.
A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once.
With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.
same root cause · shared GPU driver
same root cause · shared Kubernetes API server
same root cause · shared Linux kernel
same root cause · shared Linux kernel
same root cause · shared Linux kernel
same root cause · shared Linux kernel
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server