IL-0079 Altinity · 2025-05-17kubernetesgcpnetworking
Early on 17 May, several GKE clusters running Altinity.Cloud ClickHouse services lost all ingress traffic. GKE had auto-upgraded their worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003 with kernel 6.8.0-1022-gke. That kernel lacked support for extensions kube-proxy uses, because of a regression in how the nf_tables module was packaged or loaded, so kube-proxy could not apply its IPv6 iptables rules. Its health check failed, the load balancer health checks that depend on it failed, and the load balancers stopped sending traffic to the nodes even though pods and nodes were otherwise healthy.
The regression was a known, ongoing problem in the Ubuntu kernel series, tracked in Ubuntu bug report #2101914, a Linux kernel mailing list thread and a community write-up. Altinity names four reasons it was hard to avoid: GKE auto-upgrades nodes with no way to opt out, IPv6 iptables rules are processed even when IPv6 is not used, GKE's image validation did not catch the problem, and the newer image builds had the same regression, so there was nothing working to roll forward to.
Altinity first added maintenance exceptions to pause upgrades on the clusters that were not yet affected, which kept the problem from spreading to all of its GKE clusters. Service on the affected clusters came back once the load balancer health check policy was changed to ignore failed kube-proxy health checks, without changing workloads or infrastructure. For a lasting fix Altinity started moving the affected clusters from ubuntu_containerd to the cos_containerd image family, whose kernel does not have the broken nf_tables behaviour; this meant recreating nodes and a small amount of downtime. At the time of writing GKE had not released a fixed Ubuntu image.
On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.
The failures traced back to worker nodes that GKE had just auto-upgraded to the ubuntu_containerd image 1.32.3-gke.1785003 with kernel 6.8.0-1022-gke. Shortly after the upgrade, traffic to and from these nodes dropped to zero and the load balancer health checks against them failed.
kube-proxy logs showed it failing again and again to apply iptables rules, in particular for IPv6, with errors like "Warning: Extension MARK revision 0 not supported, missing kernel module?" and "ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"". The workloads did not use IPv6, but GKE nodes process IPv6 iptables rules by default.
Rolling forward to the next available image, 1.32.3-gke.1927002, did not help: it had the same kernel problem.
Symptom ? Service outage
Root cause ? Breaking change in an upgrade
An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.
A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause.
Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.
same root cause · shared Node operating system image
same root cause · shared Linux kernel
same root cause · shared Cloud load balancer
same root cause · shared Node operating system image
same root cause · shared Node operating system image