Terraform Wants to Replace the Database. Apply Now?
A routine terraform plan unexpectedly wants to destroy and recreate the production database. Defend whether to apply, and how.
Answer key reviewed by Branko Petric
Read first: Terraform Fundamentals for DevOps
Your decision
Your terraform plan for a small tag change shows it will destroy and recreate the production RDS instance (force replacement). It is mid-afternoon, peak traffic. Do you apply now, and if not, what do you do instead?
The situation
You open a pull request that changes one tag on your RDS instance. CI runs terraform plan. Instead of a quiet in-place update, the plan reports: aws_db_instance.main must be replaced with # forces replacement next to one attribute, and a summary of 1 to destroy, 1 to add.
Context
It is 2:40 PM on a Tuesday, near peak traffic. The instance holds live production data. Hourly automated snapshots are enabled. Three other engineers are waiting on this PR to merge so they can rebase. The CI pipeline has an apply step that is one approval click away.
Related challenges
- Someone Changed Infra by Hand. Terraform Plan Is a Mess.
A manual console change put Terraform's state out of sync, and the next plan wants to undo it. Defend how you reconcile it.
- One Giant Terraform State Runs Everything
A single Terraform state holds all your infra; every plan is slow and scary. A teammate wants to just keep applying carefully. Defend how you reduce the blast radius.
- Terraform Doesn't Know the Hand-Made DB. Let It Build a New One?
A prod RDS instance was made by hand and Terraform's plan wants to create a new one. A teammate says just apply. Defend what you actually do first.
- Commit terraform.tfstate to the Repo So Everyone Shares It?
A teammate wants to commit terraform.tfstate to git so the whole team shares one state file. Defend the correct way to share Terraform state.