IL-0024 Hugging Face · 2024-04-22ai-infra
Heavy reads on the info-cache collections overloaded the database primary. The application did not cancel database work when clients disconnected, so retries piled up into a thundering herd. The database cluster did not scale to the needed tier on its own. The team blocked one user agent at the firewall, disabled quick search, put the site into maintenance, scaled pods down and back, rate-limited aggressively and scaled the database tier by hand. Finding the root cause took two to three days; full service returned on the third morning.
The Hub became unstable in the morning and unresponsive within an hour. Two read-only database nodes kept restarting and the pages that did load were slow. The outage repeated on the following two days.
Root cause ? Software defect
Trigger A high-volume endpoint reading model and dataset revision info, including one call listing hundreds of spaces for a single model, plus repeated request patterns from one client.
Database and application metrics; a support case was opened with the cloud provider.
The team learned which metrics to watch, detailed database inspection exposed the bottlenecks, and unrelated simultaneous failures made root-cause work slow.
–