Your Primary Region Is Down. Fail Over Now?
The primary region is degraded. A teammate wants to flip DNS to the standby immediately. Defend whether and how you fail over.
Answer key reviewed by Branko Petric
Read first: AWS Databases: RDS and DynamoDB
Your decision
Your primary cloud region is having a major outage and your app is degraded. You have a standby in another region with an async database replica. A teammate says just flip DNS to the standby right now. Do you fail over, and how?
The situation
Your primary cloud region is in a major outage. The app is timing out and erroring for users. You maintain a warm standby in a second region backed by an asynchronous database replica.
Context
The primary is degraded but it is not clear it is fully dead - parts of it may still be reachable and writable. The async replica is some seconds behind. A teammate wants to immediately update DNS to send all traffic to the standby region.
Related challenges
- Add a NOT NULL Column to a 50M-Row Prod Table
A migration adds a NOT NULL column to a huge live table. A teammate wants to run the ALTER directly. Defend the safe path.
- Failover Lost 5 Minutes of Writes. Promise Zero Data Loss?
An async-replica failover dropped five minutes of committed writes. Leadership demands zero data loss with zero performance impact. Defend the honest trade-off.
- Nightly Backups Run. Are You Actually Covered?
Automated backups have run for months. A teammate says you are fully covered for disasters. Defend whether you agree.
- Every Deploy Hammers the Database for 10 Minutes. Bigger DB?
Each deploy flushes the cache and the database melts for 10 minutes while it refills. A teammate wants to double the database size. Defend fixing the stampede instead of paying for peak.
More: Database scenario interview questions for DevOps and SRE