IL-0018 UK Ministry of Justice · 2023-01-11kubernetesawssecurity
Nodes and the CI system kept logging in with the old password. The registry rate-limited the repeated failures, so every pull failed. Updating the password in Terraform would have rewritten the node launch template and forced a node recycle during working hours, so the team put the previous password back to restore pulls within an hour and a half, and planned a clean rotation with a graceful recycle out of hours. The lasting change was a dedicated registry token for the platform instead of a shared human password.
Image manifests from the public registry failed with a too-many-failed-logins error. The cluster's own health check showed nodes could not pull images, and a user reported ErrImagePull in two namespaces.
Symptom ? Image pull failures
Root cause ? Operator error
Trigger The registry password was rotated during a password-manager remediation without updating the copy baked into node configuration and the CI system.
A team member saw the registry error, the health-check DaemonSet confirmed it, and a user reported it in the support channel.
A dedicated token removes the coupling between a shared human password and node configuration.
–