IL-0173 Devsisters · 2024-02-13kubernetessecurity
Devsisters encrypts traffic in its Kubernetes environment with Let's Encrypt certificates issued through cert-manager. A Let's Encrypt RSA certificate could chain two ways: DST Root CA X3 to ISRG Root X1 to Let's Encrypt R3, or ISRG Root X1 to Let's Encrypt R3 directly. The DST Root CA X3 cross-sign, valid until 30 September 2024, was kept so that older Android devices, which do not trust ISRG Root X1 but also do not check the expiry of trust anchors, could still connect. Because the old BoringSSL in one of its libraries rejected that chain, Devsisters had set cert-manager's preferredChain to ISRG Root X1.
On 8 February 2024 Let's Encrypt swapped the two: the ISRG Root X1 chain became the default and the DST Root CA X3 chain the alternate, which an ACME server offers through a Link header (RFC 8555, section 7.4.2). Two design choices in cert-manager's chain selection then picked the wrong chain. It searches only the alternate chains, never the default one, and it compares the requested name with the issuer common name of every certificate in a chain rather than only the one closest to the root. The alternate DST Root CA X3 chain contains a certificate issued by ISRG Root X1, so preferredChain: ISRG Root X1 matched it and cert-manager served the chain through the expired root. Certbot, by comparison, checks only the issuer of the last certificate in each chain.
The engineer verified the analysis against the ACME specification, Certbot's debug log of a real exchange with the Let's Encrypt ACME server, and the code of both clients. Nothing had yet been reported in the Let's Encrypt community forum or the cert-manager repository, so the engineer sent a pull request to cert-manager, which was merged; the maintainers took care over users who might depend on the old behaviour. The same issue was reported to cert-manager a day after the pull request and shared in the Let's Encrypt community about five days later. Because the problem showed up in a development environment, it had no service impact.
On 13 February 2024, when Devsisters' infrastructure cell came back from the Lunar New Year holiday, it received a request to look into a development server that could not be reached. No one had made a manual change over the holiday.
Inspecting the certificate chain the server presented, the team found it was serving a chain anchored at DST Root CA X3, a root certificate that had expired at the end of September 2021. cert-manager had been configured with preferredChain: ISRG Root X1 precisely to avoid that chain, because a library the team used was built on an old version of BoringSSL. With its default settings, that BoringSSL fails to build a trust chain whenever it receives the chain through DST Root CA X3 and DST Root CA X3 is not valid.
Tests showed that the chain cert-manager returned depended on the preferredChain value. With no preferredChain set, it returned the ISRG Root X1 chain. With preferredChain: DST Root CA X3, it returned the DST Root CA X3 chain, as expected. With preferredChain: ISRG Root X1, it also returned the DST Root CA X3 chain.
Symptom ? TLS certificate errors
Root cause ? Software defect
On 13 February, after the Lunar New Year holiday, a request came in to the infrastructure cell that the development server could not be reached; while inspecting the certificate chain the server presented, the team found it serving the chain through the expired DST Root CA X3.
–
A latent bug can stay hidden until an upstream provider changes a default. cert-manager's preferredChain matching gave the intended result while DST Root CA X3 was Let's Encrypt's default chain and the wrong one after the default changed.
An expired root at the top of a served chain is harmless to most TLS clients, which fall back to the trusted ISRG Root X1 below it, but fatal to an implementation such as an old BoringSSL that checks the trust anchor's validity and insists on verifying the topmost certificate. ACME clients choose the preferred chain with different heuristics, so the same preferredChain value can behave differently between Certbot and cert-manager.
–