Failover Lost 5 Minutes of Writes. Promise Zero Data Loss?
An async-replica failover dropped five minutes of committed writes. Leadership demands zero data loss with zero performance impact. Defend the honest trade-off.
Answer key reviewed by Branko Petric
Read first: AWS Databases: RDS and DynamoDB
Your decision
Your primary database died and you failed over to an async replica. Five minutes of committed writes - orders customers got confirmations for - are gone. Leadership says: guarantee this never happens again, with zero data loss and zero performance impact. What do you tell them, and what do you actually build?
The situation
At 02:10 the primary database's host died. Failover promoted the async replica and the app came back at 02:24. Then support tickets started: customers have order-confirmation emails for orders that do not exist. The replica was about five minutes behind at the moment of failure, and every write in that window - roughly 1,900 transactions, including around 140 paid orders - is gone.
Context
Replication has always been asynchronous; it was set up years ago and nobody revisited it. The morning after, the CTO puts it simply: "Committed means committed. I want a guarantee this can never happen again - zero data loss - and it obviously cannot slow the product down, we compete on speed." You are asked to come back tomorrow with the plan.
Related challenges
- Your Primary Region Is Down. Fail Over Now?
The primary region is degraded. A teammate wants to flip DNS to the standby immediately. Defend whether and how you fail over.
- Nightly Backups Run. Are You Actually Covered?
Automated backups have run for months. A teammate says you are fully covered for disasters. Defend whether you agree.
- Add a NOT NULL Column to a 50M-Row Prod Table
A migration adds a NOT NULL column to a huge live table. A teammate wants to run the ALTER directly. Defend the safe path.
- Every Deploy Hammers the Database for 10 Minutes. Bigger DB?
Each deploy flushes the cache and the database melts for 10 minutes while it refills. A teammate wants to double the database size. Defend fixing the stampede instead of paying for peak.
More: Database scenario interview questions for DevOps and SRE