IL-0160 Linode · 2026-03-04kubernetesai-infra
During a routine improvement effort, Linode teams found a broken legacy GPU monitoring script that had stopped reporting correct metrics, and repaired it to restore a missing metric on internal observability dashboards. The script had been written for an earlier GPU generation and sent a firmware inspection query, something the scope of the fix did not make obvious. On the RTX 4000 Ada architecture, that kind of query against an active GPU triggers a race condition in the GPU System Processor (GSP), and the GPU goes into a protective lockup state in which running workloads can no longer use it.
The lockups began at about 21:00 UTC on March 4, 2026. At first they seemed limited to LKE worker nodes, but Linode later confirmed that all Linodes with RTX 4000 Ada GPUs were affected. A first fix did not hold and the problem came back in several regions. Linode disabled the monitoring script across the GPU fleet and rebooted the nodes, and the issue was fully mitigated at about 17:16 UTC on March 5. The postmortem notes that the investigation was still ongoing when it was posted.
From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.
The first status update, at 02:07 UTC on March 5, described a critical issue on RTX 4000 Ada GPU nodes in several regions, including Osaka (osa1), Seattle (sea1) and Chicago (ord1). Affected GPU nodes could report an unrecoverable error state, so Vulkan initialisation and GPU-accelerated workloads failed. Some LKE clusters in Osaka also had control plane connectivity problems, with API requests timing out or returning errors.
The investigation first looked at a possible regression in the host hypervisor or the GPU firmware and ruled out recent software releases. A fix was reported at 07:34 UTC, but at 10:50 UTC the problem returned across multiple regions. The actual cause was identified at 16:23 UTC.
Root cause ? Faulty deployment
Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.
A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent