IL-0021 Google Cloud · 2024-03-12kubernetesgcpai-infra

GKE GPU nodes came up without a working NVIDIA driver

GKE nodes on Container-Optimized OS fetch the NVIDIA driver at boot from a bucket owned by the driver supplier. Access to that bucket failed, so every new or recreated GPU node for T4, L4, A100 and H100 came up without a driver in every GPU region, for almost seven hours. Automatic node repair kept recreating nodes and retrying, so it was paused as a partial mitigation while the access problem was fixed with the partner. Customers with auto-repair disabled had to recreate nodes themselves afterwards.

Severity: sev2
Scope: new and recreated GPU nodes in all GPU regions
Users affected: –
Region: global
Time to detect: –
Time to resolve: 6h55m

Symptoms

Newly created or recreated GPU nodes started without a functional GPU. On the node the driver install failed with a forbidden response from the download.

Symptom ? GPU unavailable Node NotReady

Root cause

Root cause ? Dependency failure

Component ?GPU driver Node operating system image Node pool

Contributing factor ?Automation amplification

Detection

unknown

Remediation

paused automatic node recreation
fixed bucket access with the driver partner

Action items

–

Lessons

The report states no lessons beyond the description.

Sources

https://status.cloud.google.com/incidents/aRSt8sTQLKMTVgdbbK6P · official-status · archived copy

References

https://cloud.google.com/compute/docs/gpus/gpu-regions-zones

Related cases

IL-0025 Link flapping, hangs and slowdowns while bringing up a 4,088-GPU cluster
same root cause · shared GPU driver