IL-0092 UK Ministry of Justice · 2026-03-30kubernetessecurity
The MoJ Cloud Platform team was moving its ingress-nginx from version 1.12.0 to 1.14.3, and from images maintained by the Kubernetes project to images maintained by Chainguard, because the Kubernetes project was retiring ingress-nginx and its images. Version 1.14.3 bundled a newer ModSecurity web application firewall and a newer OWASP Core Rule Set with better detection of malicious traffic. The team had offered a beta ingress for testing from 12 March, told service teams about it on 13 March, deployed a release candidate to the non-production ingresses on 23 March, and published a summary of the ModSecurity and CRS changes the same day.
The rollout to the three production ingresses started at 07:11 on 30 March. The new rules produced two false positives: one on the contents of a PostHog analytics cookie and one on an Expect header sent to HMPPS Auth. The cookie problem had been found in testing, but the mitigation was not enough: the cookie was rescoped so it would no longer be sent to all *.justice.gov.uk services, yet too little time passed before the new rules went live for the old cookie to expire or be replaced in users' browsers. The Expect header problem was not found in testing by the affected service teams, and the report says it is not yet clear why. The specific rule changes behind each block were not identified.
At 09:14 the team began rolling ingress-nginx back, and the rollback completed at 09:43. HMPPS developers had added a local exclusion for the rules blocking HMPPS Auth at 09:22, and at 09:49 confirmed the elevated blocking had stopped. At 11:23 the Cloud Platform team created a temporary global exclusion for the rule triggered by the PostHog cookie, using findings from HMPPS and from LAA's testing. With both exclusions in place the update was redeployed and completed at 14:00, and at 14:39 HMPPS confirmed the problem had not returned.
At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.
The web application firewall was rejecting requests to the HMPPS Auth application as false positives. Users across HMPPS could not authenticate and so could not reach their line-of-business applications. Because of the nature of one of the false positives, users in other parts of the MoJ may also have been unable to reach applications running on the Cloud Platform.
HMPPS developers identified the OWASP Core Rule Set (CRS) rules that were blocking requests. Two payloads triggered them: the contents of a cookie used by the PostHog analytics service, and an Expect header in requests sent to HMPPS Auth.
Symptom ? High error rateService outage
Root cause ? Breaking change in an upgrade
At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.
An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them.
A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller