IL-0107 DFDS · 2022-08-03kubernetes

Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

DFDS runs Kyverno, a policy engine for Kubernetes, with a ClusterPolicy whose admission webhook checks Service resources. Kyverno had not been kept up to date because of a code freeze. When a certificate chain expired in a third-party application, Kyverno got stuck in a crash loop. The ClusterPolicy's failurePolicy was left at the default, fail, so with the webhook unresponsive the Kubernetes API server rejected Service operations instead of letting them through, in every namespace that was not exempted. With failurePolicy set to ignore, the postmortem says, the issue would not have occurred.

The platform team started looking at the report at 11:50 UTC, identified the issue at 12:08 and reproduced it in a sandbox cluster at 12:18. The fix was to update Kyverno to a newer Helm chart version; it went into the sandbox first and reached the affected production cluster at 13:06. At 13:25 the user team confirmed that it worked. The postmortem gives 12 minutes to detection and 88 minutes to recovery.

The postmortem also lists that Kyverno went into production too early, without monitoring and without enough communication and coordination beforehand, that the problem had been raised but was not on the 'ops day to day' board, and that the team saw signs of Kyverno not being a mature project.

Severity

–

Scope

create, update and delete of Service resources in all non-exempted namespaces of one production Kubernetes cluster

Users affected

end-user teams deploying to the affected cluster

Region

–

Time to detect

12m

Time to resolve

1h 28m