IL-0192 Grafana Labs · 2025-02-18kubernetesnetworking
An engineer was upgrading the TLS policy version on the cloud load balancers used by many Grafana Cloud services, including metrics and logs ingestion. The engineer tested the new configuration by hand in the development environment and confirmed it could be applied without redeploying the load balancers. Grafana Labs manages many cloud provider resources from Kubernetes with Crossplane, so the code change was made in a Crossplane composition: its merge policy was changed so that the default TLS policy could be overridden. The engineer saw the override survive reconciliation and took that as enough of a test, so the full redeployment and test in development was skipped. The pull request was reviewed by a peer before it was merged.
The new merge policy also overrode a default value from the base configuration that was never meant to change. Many Kubernetes Services switched from type LoadBalancer to type ClusterIP, and Kubernetes destroyed their load balancers. About 25% of Grafana Cloud services were affected for about 150 minutes. Grafana Labs normally rolls changes out in waves, first development, then staging, then canary and production in steps, observing each wave, typically for at least 24 hours, with what is effectively a feature flag deciding which wave gets a change. This change had no such flag and was applied manually, and nothing stopped it from reaching development, staging and production at once. Deletion protection at the Crossplane level did not help, because the Crossplane resources were not deleted; they were changed so that the objects underneath became invalid.
The team rolled back the code change and recreated the affected Kubernetes Services so that Kubernetes created the load balancers again. Most applications were fully recovered by 11:10 UTC. It was not a security incident, and no customer data was leaked or exposed.
At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.
The load balancers served many services, including the ingestion of metrics and logs. Customers lost access to Grafana Cloud services, and customers whose agents could not buffer for the length of the outage may have lost metrics or logs before they reached Grafana Cloud.
The missing load balancers traced back to a recent configuration change made while updating the TLS policies on the load balancers. The Crossplane resources still existed; what had changed were the Kubernetes Services underneath, many of which were now of type ClusterIP instead of LoadBalancer.
Symptom ? Service outageResources deleted
Root cause ? Configuration change
Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.
A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.
same root cause · shared Cloud load balancer
same root cause · shared Configuration templating
same root cause · shared Configuration templating
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Configuration templating
same root cause · shared Cloud load balancer
IL-0052same root cause · shared Cloud load balancer