IL-0103 Tetrate (unnamed U.S. federal customer) · 2025-02-26 (published)kubernetesaws
A U.S. federal customer of Tetrate runs more than 30 Amazon EKS clusters with Istio and mirrors traffic across sibling clusters, isolating a cluster from traffic while it is maintained. During an update of the node AMIs, one cluster's sidecar-injected workloads went into CrashLoopBackOff. The cluster had no active traffic, but losing a sibling cluster weakened resiliency, and strict security compliance and limited production infrastructure made a fallback cluster hard to deploy quickly. With its own team busy upgrading the other clusters, the customer raised a Severity 1 ticket with Tetrate Support; a Tetrate engineer was reading logs within six minutes, and two more joined a live debugging session within 20 minutes.
In this setup Istio offloaded the certificate authority role to an external service: each sidecar's istio-agent asked for a signed certificate, istiod passed the request to the external CA, and the CA returned the signed certificate. Because the CSRs were being processed but the sidecars still failed, the team suspected the custom CA, and found it overwhelmed by the sudden flood of certificate requests from hundreds of pods starting at once. istiod's default CA request timeout of 10 seconds made the thundering herd worse.
Tetrate engineers temporarily reduced istiod from 40 replicas to 1 to slow Istio's internal processing and ease the load on the CA, found and deleted the many CSRs stuck in the Approved, Failed state, and restarted pods gradually instead of all at once. The issue was resolved in less than an hour. The write-up says the root cause was not an Istio malfunction but the overloaded external CA.
While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.
The first sign was istiod taking unusually long to initialise, with the event `Readiness probe failed: HTTP probe failed with status code: 503` on the istiod pod. While istiod was not ready, no injector webhook was available, and pod creation failed with `Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": no endpoints available for service "istiod"`.
The istiod delay passed, but once sidecars were injected the pods still did not become ready: the sidecar process aborted before it was operational. The sidecar logs ended with `warn ca ca request failed, starting attempt 1 in 99.993416ms` and `warn Aborting proxy`, while the Kubernetes events showed the CertificateSigningRequests as signed (`The CSR has been signed`).
Symptom ? Container crash loopProbe failures
Root cause ? Dependency failure
Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.
–
When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle.
The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.
same root cause · shared Node operating system image
same root cause · shared Node operating system image