3 AM Outage: Roll Back or Fix Forward?
A deploy 20 minutes ago correlates with a spike in 500s. You are on call. Defend whether to roll back or fix forward.
Answer key reviewed by Branko Petric
Read first: Continuous Deployment with GitHub Actions
Your decision
Checkout error rate jumped from 0.1% to 9% right after release v412. You can roll back to v411 in about 2 minutes, or push a fix forward with an unknown ETA. What do you do, and why?
The situation
It is 3:07 AM. PagerDuty wakes you. The checkout service error rate dashboard shows 500s climbing from a 0.1% baseline to 9% over the last six minutes.
Context
Release v412 deployed at 02:47 AM, about 20 minutes ago. The deploy pipeline is green. You have a one-command rollback to v411, which last ran clean for nine days. You do not yet know the root cause. A teammate in Slack says "I think I can patch it, give me a sec."
Related challenges
- After an Outage, Mandate Manual Approval on Every Deploy?
After a bad deploy caused an outage, leadership wants a mandatory human sign-off on every production release. Defend a better way to prevent the next one.
- It Is Just a Config File. Push It to Every Region at Once?
A generated rules file reached every edge proxy in every region within seconds, crashed all of them at once, and now the team wants a fix. Defend how config should ship.
- The Pipeline Is Slow. Skip Tests to Ship?
A 40-minute pipeline is blocking releases. A teammate wants to skip the test stage to ship faster. Defend your call.
- A GitHub Action You Use Was Hijacked. Bump the Tag and Move On?
A third-party scanner action your repos pin by tag was hijacked for half a day. A teammate bumped the tag and wants to close the ticket. Defend what you do.