Secrets Rotation Took Down Prod at 3 AM. Pin Credentials?

Automated rotation revoked a database password while pools and caches still held it, locking prod out at 3am. A teammate wants to pin credentials for a year. Defend fixing rotation instead.

advanced13 min4 roundsSecurity

Answer key reviewed by Branko Petric

Your decision

Your new automated secrets rotation rotated the database password at 3am. Connection pools and config caches across a dozen services kept using the old credential, authentication failed platform-wide, and prod was down for 90 minutes. A teammate says rotation is clearly too risky - pin the credentials and rotate manually once a year. What do you do, and why?

The situation

At 03:04 the new automated secrets-rotation job rotated the production database password: it set a new password on the database user and updated the secret in the secrets manager. Within a minute, every service that talked to the database began failing authentication - their connection pools were built with the old password at startup, and several services also cache config for 15 minutes. Prod was down for 90 minutes while on-call manually reset the password and restarted services.

Context

Rotation was rolled out fleet-wide on its first production run, overnight, with no auth-failure alerting tied to it. In the postmortem a teammate argues: "This is the second-worst outage of the year and it was entirely self-inflicted. Rotation is too risky for the database credential. Pin it, put a calendar reminder to rotate manually once a year, and move on." Security had originally pushed for rotation after an audit finding. You own the recommendation.

More: DevSecOps scenario interview questions