IL-0186 Hans H. Kim · 2023-04-13 (published)kubernetesnetworking
Components installed for storage work automatically added iptables rules to one node. Packets sent from pods are handled by the nat PREROUTING chain, so every HTTP request from a pod on that node to an external address was redirected to local port 47080, and every HTTPS request to local port 47443. Pods on the node therefore had no outbound HTTP or HTTPS. Traffic from the host itself, such as image pulls, kept working, because the matching OUTPUT rules applied only to traffic going out through the loopback interface.
The author's team asked the people responsible for the storage components to delete the rules, which fixed it.
On one worker node of a Kubernetes cluster, traffic to ports 80 and 443 went missing: pods on that node could not make outbound HTTP or HTTPS requests. Infrastructure engineers and the author's teammates debugged it.
Listing the nat table on the node (test-a100) with iptables -L -t nat -v showed two REDIRECT rules after the KUBE-SERVICES rule in the PREROUTING chain: "tcp dpt:http redir ports 47080" and "tcp dpt:https redir ports 47443". The OUTPUT chain held the same two redirects, but only for traffic leaving through lo.
Image pulls on the node still worked. They are made by the host, not from inside a pod, so they go through the OUTPUT chain, where the redirects only matched traffic to localhost.
Symptom ? Misrouted requests
Root cause ? Configuration change
–
–
Inside Kubernetes, kube-proxy itself manages iptables rules, so the author felt that part needs to be understood in depth. When packets go missing during network tests, check each step of the path in turn, which requires knowing clearly which steps the networking goes through.
–