Every Deploy Hammers the Database for 10 Minutes. Bigger DB?
Each deploy flushes the cache and the database melts for 10 minutes while it refills. A teammate wants to double the database size. Defend fixing the stampede instead of paying for peak.
Answer key reviewed by Branko Petric
Read first: AWS Databases: RDS and DynamoDB
Your decision
Every deploy flushes your Redis cache, and for the next 10 minutes the database runs at 95% CPU serving a flood of identical queries while the cache refills. Users see timeouts. A teammate proposes doubling the database instance size so it can absorb the refill. Do you upsize the database, or fix something else? Defend your call.
The situation
Your team deploys a mid-sized e-commerce API three or four times a day. The deploy script flushes the entire Redis cache "to avoid stale data", and for roughly 10 minutes afterward the primary Postgres instance sits at 90-95% CPU while the cache refills. During that window, p99 latency goes from 80ms to 6 seconds and a few percent of requests time out. Support has started asking why the site "breaks every afternoon".
Context
The database is an r6g.xlarge that runs at 25-30% CPU outside the refill window. Cache hit rate is normally around 97%; product and pricing queries are the expensive ones being duplicated during refill. There is no request coalescing anywhere in the stack, TTLs are a uniform 15 minutes, and the deploy pipeline runs a plain FLUSHALL. A teammate has already priced the next instance size up and posted in the channel: "double the DB and this whole problem disappears, it is only money."
Related challenges
- Nightly Backups Run. Are You Actually Covered?
Automated backups have run for months. A teammate says you are fully covered for disasters. Defend whether you agree.
- Add a NOT NULL Column to a 50M-Row Prod Table
A migration adds a NOT NULL column to a huge live table. A teammate wants to run the ALTER directly. Defend the safe path.
- Your Primary Region Is Down. Fail Over Now?
The primary region is degraded. A teammate wants to flip DNS to the standby immediately. Defend whether and how you fail over.
- Failover Lost 5 Minutes of Writes. Promise Zero Data Loss?
An async-replica failover dropped five minutes of committed writes. Leadership demands zero data loss with zero performance impact. Defend the honest trade-off.
More: Database scenario interview questions for DevOps and SRE