IL-0054 DevOps Hof · 2019-02-01 (published)kubernetesgcpnetworking
Three Kubernetes behaviours combined. First, the ingress-nginx manifest for GCP sets externalTrafficPolicy: Local on its Service by default, which keeps the client source IP but means traffic is only delivered to ingress-nginx pods on the node that received it, never forwarded to another node. Second, with that policy Kubernetes creates a GCP HTTP health check on the healthCheckNodePort's /healthz, which kube-proxy deliberately fails on every node that has no local ingress-nginx pod. Third, the Kubernetes service controller only puts nodes that are Ready and schedulable into the load balancer backend, so a cordoned node is removed as soon as the drain starts.
During the node pool migration the old nodes were cordoned and dropped out of the backend, but the ingress-nginx pods stayed on them because pod disruption budgets blocked their eviction. The backend now held only new nodes, all failing the health check because no ingress-nginx pod ran there, so every ingress on the cluster stopped receiving traffic. Deleting the leftover nginx-ingress-controller pods on the old nodes let them be scheduled on the new nodes and restored traffic.
While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.
The LoadBalancer service of ingress-nginx had changed the backend pool of the GCP load balancer so that it pointed only at the new nodes. The nginx-ingress-controller pods, however, were still running on the old nodes, because problems with pod disruption budgets had kept them from being evicted. The load balancer health checks against the new nodes all failed.
The team first expected kube-proxy to forward the traffic to the right node, since it normally sends non-local traffic on to wherever the pods run, so the question was why that did not happen here.
Symptom ? Service outageProbe failures
Root cause ? Operator error
The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.
–
With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.
same root cause · shared Ingress controller
same root cause · shared Node pool
same root cause · shared Ingress controller
same root cause · shared Ingress controller