IL-0085 Magic Pages · 2025-04-14kubernetes
Magic Pages hosts Ghost websites on a Kubernetes cluster on Hetzner Cloud, built with the hetzner-k3s tool, with three control plane nodes for redundancy. When one control plane node ran out of memory, the loss of one node should have been absorbed by the other two. The cluster, however, depends on the Hetzner API for ongoing work such as networking and some storage management, and with the control plane in trouble these components kept calling the API. They used up the project's limit of 3600 requests per hour and kept it at zero, so the failed node could not be replaced and the existing cluster components could not work either, and every customer website went down. The founder knew the limit existed but had not realised how much the running cluster itself relied on the API. Two days later Hetzner told the founder that a bug in a cloud controller that connects Kubernetes clusters to Hetzner Cloud was involved; its root cause had not been found for several months, and Hetzner provided a workaround for faster recovery next time.
To stop the API calls, the founder powered off every server in the cluster. The ratelimit-remaining value then started to climb, and the founder waited longer to build a buffer and asked Hetzner support for a higher limit. The control plane servers were restarted successfully. When the worker nodes came back, every website container tried to start at the same moment, but the sites need their Longhorn storage volumes first, and Longhorn needs a minimum number of its own components running after a full shutdown. These essential Longhorn components had to be started by hand; after that Longhorn healed itself and made the volumes available over about an hour, and the website containers started as their volumes appeared.
By about 17:00 UTC around 95% of the sites were back; the rest, including the founder's own blogs, needed manual restarts shortly after. The outage lasted about 7.5 hours. The CDN, when activated, served cached copies of sites, and the databases were unaffected because they had been moved out of the Kubernetes cluster earlier in the year.
At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.
The usual fix is to remove the failed node and create a replacement, so the founder started a replacement on a slightly larger server with the hetzner-k3s tool. The deployment failed, and a second attempt failed too. hetzner-k3s gave no detailed log of why the underlying Hetzner API calls were failing. At about the same time monitoring alerts began to flood in because customer websites were becoming unreachable.
After about half an hour of investigation the cause turned out to be the Hetzner Cloud API rate limit of 3600 requests per hour per project. Waiting for the hourly limit to reset did not help: after 30 minutes the API still reported ratelimit-remaining: 0. It took another 15 to 20 minutes to see why. The remaining Kubernetes nodes, struggling with the failing control plane and the lack of API access, kept calling the API, so the quota never refilled. New nodes could not be created, and moving traffic to a separate cluster was not possible either, because that also needed API calls.
Symptom ? Service outageNode NotReadyCloud API rate limit exhausted
Root cause ? Capacity exhaustion
The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.
The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored.
A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.
same root cause · shared Persistent volumes and stateful sets