IL-0013 loveholidays · 2020-02-15 (published)kubernetesgcpnetworking
The nodes' connection-tracking tables filled up. On GKE the table size scales with node memory, roughly five entries per megabyte, which is not what the commonly cited formula predicts, so shrinking the nodes had shrunk the limit at the same moment the connection count doubled. Moving the workload to nodes with more memory and limiting pods per node resolved it. The team added an alert on conntrack utilisation.
After a service change and a node-pool resize, the aggregation proxy showed climbing error rates and HAProxy logged connection resets that kept increasing.
Symptom ? Connection resetsHigh error rateIntermittent connection failures
Root cause ? Capacity exhaustion
Trigger A change in service architecture, halving the memory of the aggregation nodes and sharding the downstream services, which doubled the number of tracked connections.
Connection-reset errors in HAProxy, then node-exporter conntrack metrics compared with the limit.
The documented conntrack formula did not match GKE's behaviour; node sizing directly sets conntrack capacity.
IL-0016 Prometheus on the live cluster killed by memory exhaustionsame root cause · shared Node pool
IL-0015 A GKE pod address range ran out at 256 nodessame root cause · shared Node pool