IL-0159 Linode · 2025-02-05kubernetesnetworking
On 5 February 2025 at about 21:38 UTC, Docker Hub began rate-limiting image pulls from Linode's IPv6 address space in multiple regions. Image pulls over IPv6, authenticated or not, received 429 Too Many Requests, which broke image pulls for LKE, LKE-E and other services that depend on Docker Hub.
Docker Hub explained that the traffic had exceeded its abuse limit, and that the limit was triggered by the way its IPv6 checks look at the first 64 bits of an address. As an interim fix, Linode changed its DNS resolvers to return an A record instead of an AAAA record, so image pulls to Docker Hub went over IPv4. This mitigated the problem at about 04:30 UTC on 6 February 2025. When the report was posted on 28 February, Linode was still looking for a long-term solution.
From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.
LKE and LKE-E clusters, along with other Linode services that pull images from Docker Hub, were affected. The first status post, at 22:48 UTC, described rate limiting between certain Linode data centers and Docker Hub over IPv6; the component listed on the status page was the US-ORD (Chicago) Linode Kubernetes Engine.
Symptom ? Image pull failures
Root cause ? Dependency failure
The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.
A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.
–