IL-0177 LINE+ · 2022-10-14 (published)kubernetesnetworking
messaging-hub keeps WebSocket connections to the Demaecan Merchant and Driver apps, behind an L7 load balancer from LINE's internal IaaS, Verda, followed by Ingress Nginx on Kubernetes. A colleague with SRE experience set up HAProxy, the software behind the L7 load balancer, in local Docker and sent it a HUP signal with "docker-compose kill -s HUP haproxy". After a delay the existing connections were closed: when HAProxy reloads or restarts, the old process ends after the hard-stop-after time and takes its open connections with it. The infrastructure team confirmed hard-stop-after was 30 minutes, and deploying the team's load balancer in the Verda dashboard reproduced it in production: the HAProxy uptime reset exactly 30 minutes later and the WebSocket drop happened at the same moment.
The team rarely changed its own load balancer, but the HAProxy was shared with other services, so any of them deploying its own load balancer configuration reloaded HAProxy and, 30 minutes later, cut messaging-hub's connections. That explains why the drops were irregular. Because the Verda dashboard is split by project, the team had not imagined the load balancer was shared, and had trusted the infrastructure team's answers and limited the search to systems it controlled.
Before the cause was found, the reconnect waves were contained in two places. The Merchant system moved the token issuance API into a separate deployment so token traffic would not slow other Merchant APIs. Nginx was scaled out instead of changing the conntrack limit or unloading the module, with pod anti-affinity and a label selector so each Nginx pod ran on a different node, since nf_conntrack is per node; after that, further drops no longer caused Nginx packet loss. The drops themselves ended when messaging-hub moved from the L7 load balancer to an L4 load balancer.
The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.
After each drop, clients reconnected as agreed, and each reconnect needed a new connection token issued through the Merchant or Delivery system. The token requests piled onto the Merchant system, which saw far more drops than the Delivery system, and a stress test showed its database was the bottleneck. The delay spread to other Merchant components and reached Merchant users directly. On the messaging-hub side, Nginx began dropping packets at the traffic peaks because the nf_conntrack table reached its limit.
The search for the cause of the drops ruled out many things. Adding 50% more connection-manager instances changed nothing; GC and memory were fine; the server-side disconnect logic ran only in rare cases; idle timeouts were 3 minutes in connection-manager and 5 minutes in the load balancer and Nginx, and per-connection timers could not explain simultaneous drops. About 28,000 local ports were available, TIME_WAIT counts were negligible, the open files limit was 1048576, and Nginx worker limits were above current use. Most dropped clients were Merchant apps, and a reset metric shared by the infrastructure team was first misread as clients closing connections. No Japanese carrier outage was found.
Ingress Nginx's proxy-read-timeout and proxy-send-timeout were set to 300s; cutting them to 3s in development reproduced drops, so both, and the load balancer timeout, were raised to 3600s as the Ingress Nginx documentation advises for WebSockets. The drops still happened. The infrastructure and network teams had answered several times that nothing was wrong on their side.
Symptom ? Connection resetsDatabase overloadIntermittent connection failuresHigh latency
Root cause ? Configuration change
The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.
–
A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days.
A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.
same root cause · shared HAProxy, Cloud load balancer
same root cause · shared Ingress controller, Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer