Secrets Rotation Took Down Prod at 3 AM. Pin Credentials?
Automated rotation revoked a database password while pools and caches still held it, locking prod out at 3am. A teammate wants to pin credentials for a year. Defend fixing rotation instead.
Answer key reviewed by Branko Petric
Your decision
Your new automated secrets rotation rotated the database password at 3am. Connection pools and config caches across a dozen services kept using the old credential, authentication failed platform-wide, and prod was down for 90 minutes. A teammate says rotation is clearly too risky - pin the credentials and rotate manually once a year. What do you do, and why?
The situation
At 03:04 the new automated secrets-rotation job rotated the production database password: it set a new password on the database user and updated the secret in the secrets manager. Within a minute, every service that talked to the database began failing authentication - their connection pools were built with the old password at startup, and several services also cache config for 15 minutes. Prod was down for 90 minutes while on-call manually reset the password and restarted services.
Context
Rotation was rolled out fleet-wide on its first production run, overnight, with no auth-failure alerting tied to it. In the postmortem a teammate argues: "This is the second-worst outage of the year and it was entirely self-inflicted. Rotation is too risky for the database credential. Pin it, put a calendar reminder to rotate manually once a year, and move on." Security had originally pushed for rotation after an audit finding. You own the recommendation.
Related challenges
- A Contractor Needs DB Access for a Week. Open the Port?
An external contractor needs a week of database access. A teammate wants to open the DB port to the internet with a strong password. Defend the safe alternatives.
- A Container in Prod Is Mining Crypto
A pod is pegging CPU running a crypto miner. A teammate wants to kubectl delete it and move on. Defend the real response.
- Chasing a Bug: Flip Production to DEBUG and Log Everything?
To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.
- A GitHub Action You Use Was Hijacked. Bump the Tag and Move On?
A third-party scanner action your repos pin by tag was hijacked for half a day. A teammate bumped the tag and wants to close the ticket. Defend what you do.