IL-0087 Fyra Labs · 2023-04-09kubernetesnetworking
Fyra Labs runs an RKE2 Kubernetes cluster with Multus, a meta-CNI (a CNI plugin that calls other network plugins), together with Calico, used to add extras such as bandwidth limits and network metering policies. Unusual CPU use on a node, presumed to be an upstream containerd and runc bug, led the team to cold-reboot the affected nodes. After the reboot Multus tried to set up IP addresses and virtual network interfaces again but could not reach the Kubernetes API, so the pods on those nodes got no IP address and failed.
Believing the failure was temporary, the team upgraded and cold-rebooted more nodes, and each one lost networking the same way until the whole cluster was isolated and all services were down. They uninstalled Multus and Calico completely and replaced them with the Cilium CNI, which RKE2 fully supports, then rolled back most other cluster settings. All services were back by 14:30 ICT. The write-up states that the cause is still unknown.
The team had earlier passed over Cilium because of its resource footprint in a published CNI comparison, but in their small cluster Cilium stayed within their threshold of one core per node, as Calico, Canal and Flannel had.
At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.
At around 2:30 ICT services on the rebooted nodes showed problems: the nodes could not connect to the service mesh, and no pod on them could get an IP address. Every service scheduled on those nodes failed to restart: HashiCorp Consul, HashiCorp Vault in high-availability mode on the Consul backend, the Grafana LGTM stack with Loki, Tempo and Mimir (which also use Consul), Mastodon, Traefik (partly), Terra (partly, with package uploads disabled), Subatomic, Madoguchi (a metadata service for Subatomic), KeyDB, PostgreSQL, the Hetzner Cloud CSI service and the JuiceFS CSI service.
The pods got no addresses because Multus, the meta-CNI the team ran together with Calico, could not contact the Kubernetes API through its own proxy, which had also failed to start. The log showed: "Final result of CNI DEL was an error. error=error getting ClusterInformation: Get "https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.43.0.1:443: i/o timeout".
The team assumed the problem would clear by itself. Instead, system upgrades and cold reboots of other nodes, done in the hope of re-initialising the network, brought the same failure to those nodes until every node was cut off from the network. The Terra repositories still seemed up, so the team left them as they were; at 6:31 ICT developers reported that the Terra repositories had gone down, and the incident became a major one.
Symptom ? Sustained high node CPUService outage
Root cause ? Not stated
The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.
–
Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.
–