IL-0084 Namespace · 2024-10-22kubernetesnetworking
Namespace runs customer jobs on bare metal machines, both its own hardware and machines rented from bare metal providers, managed through Kubernetes clusters tuned for fast pod creation. Growth outpaced planning towards the end of Q3 2024, and new hardware due in a European region by 15 October was delayed three to four weeks because the supplier had ordered the wrong DIMMs. Namespace ordered extra capacity from an existing provider whose machines are connected in one L2 segment by a feature the provider calls vSwitch.
From 17 October packets through the vSwitch dropped sporadically, and by 21 October jobs were failing. Without a prompt fix from the provider, Namespace moved fra3 off the vSwitch: each provider machine was connected to the nearby Namespace-managed zrh1 cluster over Wireguard, and all customer traffic was diverted from fra3 to other regions. Because the infrastructure is immutable, every machine had to be repaved with a new identity, certificates and encryption-at-rest keys, and regional caches were dropped to recover faster. Image distribution to many machines at once was slow because a few distributor machines are not built for that, so the team cloned datasets in a more distributed way. The outage ran from 12:00 to 18:30 CET on 22 October, and the rebuild was mostly complete by 20:30.
On 23 October one zrh1 control plane node, an older machine type than in other regions, started failing partially because of slow NVMe writes. The rate of change in Kubernetes had grown past what its I/O could handle, and the API server's CPU went very high from retries, so other parts of the system suffered. Workloads were redirected away from zrh1, and the node's key state was cloned to a higher-capacity node that took over its role with the same IP address; after working through pending changes the cluster returned to normal and each kubelet reconnected. That second outage lasted roughly 50 minutes.
From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.
The drops grew towards the weekend, and on Monday 21 October the SLO fell sharply and the packet drops were making customer jobs fail. The provider confirmed a problem but did not fix it promptly.
When fra3 was taken out of service on 22 October to rebuild its network, global capacity ran short, linux/arm64 first. By 16:00 CET queue times were visible across the whole fleet and some remote builds failed for lack of capacity.
On 23 October at 20:10 CET the zrh1 region began to degrade, and at 20:14 its core failed and workloads sent to zrh1 failed to start. One control plane node had backed-up I/O from slow NVMe writes and the Kubernetes API server showed very high CPU from retries. Coming straight out of a network outage, the team first looked for network-related causes.
Symptom ? Intermittent connection failuresHigh latencyControl plane lagService outage
Root cause ? Cloud provider outage
SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.
Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage.
After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.
–