Incident Response for DevOps: How to Fix What Breaks

Why incident response comes after monitoring and security

You need evidence to debug an outage. Step 10 gave you the dashboards and alerts. Step 11 showed you what a leaked key looks like and what to do about it. This step turns those pieces into one habit you can use under pressure.

It also comes before Kubernetes on purpose. Your app still runs on a setup you built yourself, so you know every part that can fail. That makes it easier to practice the method before adding more layers.

What to learn, in order

1. Scope the problem. Ask who is affected, what they see, and when it started. A failure for one user and a failure for everyone point to very different causes.
2. Check recent changes first. Look at the last deploy, the last Terraform apply, and git log. Many incidents follow a change, and this check takes a minute.
3. Follow a request through the system. Go from DNS to the load balancer to Nginx, the app and the database. curl -v shows you how far a request gets.
4. Read logs and metrics together. Metrics tell you when something changed. Logs often tell you why.
5. Test one hypothesis at a time. Write down what you think is wrong and what would prove it. If you change three things at once, you will not know which one helped.
6. Stop the harm before you find the root cause. Mitigation (a quick step that reduces the damage) comes first. A rollback (going back to the last working version) is the common example. Know how to do yours from step 7.
7. Communicate. Post short updates with what you know, what you are trying, and when you will update next. Keep a timeline as you go, because you will need it later.
8. Write the postmortem. A postmortem is a written record of what happened and what will change. Google's SRE book explains the blameless approach. Its workbook chapter adds more on how to do it well.

What to skip for now

• Setting up on-call tools. On-call means being the person who answers alerts during set hours. You can learn the paging tool on the job.
• Chaos engineering platforms, which inject failures automatically. Breaking your small app by hand teaches the same lesson at this stage.
• Incident commander training, which teaches how to lead a large response across many teams.

Common mistakes

• Restarting everything first. A restart may bring the service back, but it can also wipe the evidence. Save the logs first, unless users are being harmed and the restart is the fastest safe fix.
• Debugging while users wait. If the last deploy is the likely cause, roll back first. Finding the bug can wait until the site is back.
• Trusting the first theory. "It must be the database" feels convincing until you look. Find evidence that could prove your theory wrong.
• Writing "be more careful" as an action. It changes nothing. Write something you can check later, such as "add a test that blocks this config from merging".
• Naming a person as the cause. Ask why the system let one mistake reach users. That question leads to a fix.

Your project

Ask a friend, or write a script, to break your app three ways with no hints. Good failures include a wrong database password, a full disk, and a firewall rule that blocks a port. Time each fix and keep your notes. Write a postmortem for each failure and put them in your Git repo. Two of them will go into your portfolio.