IL-0037 Datadog · 2019-05-23 (published)kubernetesgcpnetworking
This is one of the failures in the talk "10 Ways to Shoot Yourself in the Foot with Kubernetes" by two Datadog infrastructure engineers, whose clusters ran at 1,000 to 2,000 nodes. A permissions change on a storage bucket stopped one application from starting. The application ran as a DaemonSet on about a thousand nodes, and it went into a crash loop; because its pull policy was Always, each restart was a new image pull. All traffic from the region to the image registry left through three NAT instances, so the registry saw an unacceptable pull rate from just three source IPs and rate-limited them with HTTP 429. From then on no node in the region could pull any image: running workloads continued, but new pods and jobs could not start.
The team replaced the affected NAT instances to get new source IPs, which ended the rate limiting. They had forgotten that some critical services were reached over the internet through those IPs, among them a Cloud SQL database that at the time was not available over a private VPC. Every application using that database lost its connections. After a couple of hours of work on the static IPs and the firewall rules, everything came back.
As a follow-up they added an admission webhook that disallows floating image tags, starting with latest, and attached it to workload types and to pods. Most pods are created by controllers inside the cluster, not by users, so pods for existing workloads were rejected silently and nothing new could be scheduled. Applying the webhook only to workload objects fixed it; the speaker puts the return to normal at several days to a week after the registry incident.
Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.
The image pull metrics showed a sudden jump in the number of pulls, sustained for a while, and then a second jump. The image registry was answering with a large number of HTTP 429 responses: it was rate-limiting Datadog. At that point no node in the whole region could pull from the registry.
Replacing the NAT instances stopped the rate limiting, but it broke something else: applications that reached a critical Cloud SQL database over the internet lost their connections and stopped working, and it took a couple of hours to restore them.
The follow-up fix caused more trouble. An admission webhook that rejected floating image tags was attached to pods as well as workload objects, so pods that controllers in the cluster created for existing workloads (after node failures or scale-ups) were rejected with no user to see the error, and nothing new could be scheduled. Things were back to normal several days to a week after the registry incident.
Symptom ? Image pull failuresService outage
Root cause ? Capacity exhaustion
Users reported that their jobs were not starting; monitoring had not caught it first.
The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database