SRE Incident Response Interview Questions: The Decisions Interviewers Probe

Branko Petric
Branko Petric

- 13 min read

The short answer

SRE incident response interview questions test decisions, not definitions. Interviewers want you to scope impact, check recent changes, mitigate before chasing the root cause, and defend trade-offs such as rollback vs fix forward, retries vs backoff, and failover vs waiting. Narrate each step, say what each result proves, and finish with a blameless postmortem.

The standard incident response framework is detect, triage, mitigate, postmortem. It is correct, and it will not carry you through the round on its own, because knowing the phases does not tell you what each choice costs.

The incident round is scored on decisions: when to stop investigating and act, whether to roll back, whether to fail over, what to page on. This post works one full "the service is slow" prompt with the questions, commands and narration, then covers the decisions interviewers push on most.

For the whole loop and the generic scenarios (full disk, CrashLoopBackOff), read how to pass a DevOps interview first.

What the incident round is really testing

The framework is the easy part. Interviewers listen for whether you put users first, whether each command tests a hypothesis, whether you can name what your choice costs, and whether the people around you would know what is happening. Expect pushback on whatever you pick: "Are you sure it was the deploy?" Change your mind when they give you a new fact, and hold your position when they only give you pressure.

Worked example: "One of your web services is slow"

A typical prompt is one line: "You've started seeing random complaints online that one of your web services is slow." That is all you get. The interviewer reveals the rest only when you ask the right questions. The same method works for a production engineer troubleshooting interview, which tends to go further down into Linux and networking.

Step 1: Scope before you touch anything

Do not open a terminal yet. Ask questions:

  • "Which service, and which endpoints? Is it the whole site or one page, like checkout or search?"
  • "Is it everyone, or some users? One region, one platform, one customer tier?"
  • "When did it start, and is it getting worse?"
  • "Are we seeing errors too, or only slowness?"

Narrate why: "Random complaints online are a weak signal. Before I treat this as an incident, I want to confirm it in our own data and know how many users it affects. That decides how urgently I act and who I pull in."

If your metrics confirm real impact, say you would declare an incident and split the roles: one person coordinates, others investigate. The SRE book chapter on managing incidents describes this split.

Step 2: Check the golden signals

The four golden signals are latency, traffic, errors and saturation.

If the service exposes a Prometheus histogram, start with tail latency, not the average:

# p99 latency for the service over 5-minute windows
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="web"}[5m]))
)

Then break it down to find where it lives:

# p99 per route, to see if one endpoint owns the slowness
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="web"}[5m]))
)

# error ratio
sum(rate(http_requests_total{job="web", code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="web"}[5m]))

# traffic
sum(rate(http_requests_total{job="web"}[5m]))

Your metric names will differ. What matters is the shape: rate() over the _bucket series, summed by (le) plus any dimension you split on, then histogram_quantile. Drop le from the by clause and the quantile cannot be computed.

Narrate what each result would tell you: "If p50 is flat and p99 jumped, a subset of requests is slow, which points at a specific dependency, a lock, a cold cache or GC pauses, not a general capacity problem. If traffic doubled at the same moment, I am looking at load. If errors rose with latency, timeouts are probably turning into failures, and this is now more urgent."

Then saturation. On Kubernetes:

kubectl top pods -n web --sort-by=cpu
kubectl get hpa -n web
kubectl get events -n web --sort-by=.lastTimestamp | tail -20

CPU throttling is a common source of tail latency that average CPU hides:

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="web"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="web"}[5m]))

Step 3: Ask what changed

Interviewers often plant a change, so ask for it directly: "Was there a deploy, a config change, a feature flag flip, a dependency upgrade, or an infrastructure change around when this started?"

kubectl rollout history deployment/web -n web
kubectl describe deployment web -n web | grep -i image
git log --since="6 hours ago" --oneline

Also ask about changes you cannot see in your own repo: a database maintenance window, a CDN or DNS change, another team's deploy. A generated config file that every server hot-reloads will never show up in kubectl rollout history, which is exactly why it is easy to miss (more on that below).

Step 4: Mitigate, then keep digging

Suppose the interviewer says: "A deploy went out 25 minutes ago, and p99 on /search went from 300ms to 4 seconds right after." Now say your decision and the condition attached to it:

"The timing lines up and users are affected, so I want to roll back now and investigate after. Before I do, I check one thing: did this release run a database migration that the old version cannot handle? If not, I roll back, watch p99 return to baseline, and keep the bad version's logs and traces for the investigation."

If the interviewer says nothing changed, you go to traces. Pull the slowest requests (filter traces by duration, not a random sample) and find which span owns the time. Say what you are looking for: a slow query, a saturated connection pool, a downstream call that started timing out, retries inside a span. This is the core of the p99 latency case: the service with the highest CPU is often not the one making requests slow, and scaling it moves nothing.

Step 5: Close it out

Confirm recovery with the same queries holding at baseline for a sustained period, not one good data point. Post a resolution update, then name the follow-ups: the postmortem, and the alert or test that would have caught this earlier.

Practice this one: /challenges/p99-latency-tracing

Mitigation before root cause

This decision shows up in almost every prompt. The managing incidents chapter puts restoring service ahead of finding the cause.

The strong answer is not "always mitigate first and think later". It is: mitigate with the cheapest action that is likely to help and easy to reverse, while preserving evidence. Rolling back, disabling a feature flag, draining a bad node or shifting traffic away from a broken zone all fit. Restarting everything and hoping does not, because it destroys the state you need.

A useful line: "I am trading some certainty about the cause for less user impact, and I can get the certainty back afterward from logs and traces."

Rollback vs roll forward

Both answers can be right here, so the interviewer wants to hear which facts decide it.

Roll back when a recent change is the likely cause, the previous version is known good, and rolling back is fast and safe. It gives you a bounded recovery time, which a fix with an unknown ETA cannot. Fix forward when rollback cannot undo the damage, or when the fix is tiny, already understood and ships through your normal pipeline faster than a rollback would.

The fact that is easy to forget: database migrations. If the new release ran a forward-only or destructive migration, rolling back the code can break the old version against the new schema, or corrupt data. Mention it before the interviewer does, and name expand-and-contract migrations as the way to keep rollback possible.

Expect pushback that sounds reasonable: "a teammate says they can patch it in a minute", or "the team spent weeks on this release". Neither changes the user impact. The case has more traps than that, so argue it out there.

Practice this one: /challenges/rollback-vs-forward-fix

Retry storms, backoff and jitter

A classic follow-up: "A downstream service blipped for 30 seconds, but the platform stayed down for two hours. What happened?"

Usually retry amplification. With 3 attempts at each of 3 layers, one user request can become up to 27 calls to the struggling dependency. It recovers into a wall of retries and cannot climb out, so the outage outlives its trigger (a metastable failure).

If the interviewer proposes raising retry counts so requests "eventually succeed", say no and explain why: it raises the worst case to 5 x 5 x 5 = 125 attempts and makes the exact failure worse. The fix:

  • Retry at one layer only, and pass errors up everywhere else.
  • Use capped exponential backoff with jitter, so retries do not arrive in synchronized waves.
  • Enforce a retry budget, so retries stay a small, bounded fraction of traffic.
  • Add circuit breakers that fail fast and probe for recovery.
  • Shed load at the front door so the service finishes some work instead of timing out on all of it.
  • Propagate deadlines, so nobody retries a request whose caller already gave up.
  • Retry only idempotent operations. On a payment path, a blind retry can charge someone twice.

A minimal backoff with "full jitter" looks like this:

import random, time

# TransientError stands in for whatever your client raises on a retryable failure
def call_with_backoff(fn, attempts=3, base=0.1, cap=2.0):
    for attempt in range(attempts):
        try:
            return fn()
        except TransientError:
            if attempt == attempts - 1:
                raise
            time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))

The AWS Architecture Blog post on exponential backoff and jitter compares the jitter variants, and the SRE book chapter on addressing cascading failures covers retry budgets and deadline propagation.

The same reasoning covers two related prompts. "The queue backlog is doubling, 10x the consumers?" If consumers block on a shared database or rate-limited API, 10x consumers means 10x load on that bottleneck. Compare enqueue and dequeue rates and look for poison messages first. "Every deploy hammers the database" is a cache stampede: fix the flush and coalesce concurrent misses rather than buying a bigger database.

Practice this one: /challenges/retry-storm-meltdown, /challenges/queue-backlog-growing, /challenges/cache-stampede-after-deploy

Scaling is not always the mitigation

"Autoscaling was on and a spike still took us down. Raise maxReplicas?" The ceiling was not the problem, reaction time was. CPU lags real load, pods that take a minute to start cannot catch a sudden spike, and a full cluster has nowhere to put them. Scale on a leading signal (requests per second, queue depth), cut startup time, keep node headroom, and pre-scale for known events.

Practice this one: /challenges/autoscaling-spike-outage

When to fail over regions

"The primary region is degraded. Do you fail over?" The weak answer is "flip DNS to the standby". DNS does not promote a database or stop the old primary from taking writes.

The strong answer treats it as a stateful decision driven by RTO (how long you can be down) and RPO (how much data you can lose):

  1. Confirm the primary's write path is really unsafe, not just slow.
  2. Check replication lag on the async replica. That is the data you lose by promoting now, so make it an explicit, accepted decision.
  3. Fence the old primary so it cannot keep accepting writes. A half-down region that still takes writes gives you split-brain: two primaries that diverge, which is much harder to fix than the outage.
  4. Promote the replica, then move traffic.
  5. Plan the fail-back and the reconciliation before you need them.

Also say when you would not fail over: if the provider expects recovery soon and losing recent writes is worse for the business than more downtime. Waiting can be correct. Waiting by reflex is not.

Practice this one: /challenges/region-failover-dr

Global changes and blast radius

"A config file went to every server at once and crashed all of them. Manual approval, a bigger limit, or a faster push?" None of them. Treat config and generated data like code: validate before publish, roll out to a small slice first with automatic halts, keep a one-command revert, and make consumers keep running on the last good file instead of crashing. Cloudflare's 18 November 2025 outage had this shape: a ClickHouse permissions change made a query return duplicate rows, which roughly doubled the Bot Management feature file. The file went to the whole network, went past a 200-feature limit in the proxy, and the proxy's Bot Management module panicked and returned 5xx errors.

Practice this one: /challenges/global-config-push-instant

What to page on: alert fatigue

"On-call gets paged at 3am by a CPU alert, and nothing is ever wrong. Do you silence it?" Neither silencing it nor keeping it is right.

The problem is that the alert pages on a cause (high CPU) instead of a symptom users feel. Page on what hurts users or threatens the SLO: error rate, latency, and saturation that really predicts failure. Every page should be urgent and need a human to do something now. The rest belongs on a dashboard or in a ticket. Demote the CPU alert to a warning rather than deleting it, so you are not blind the day CPU really is the cause. Raising the threshold to 95% keeps the same design flaw.

The SRE book's monitoring chapter argues for symptom-based paging, and the workbook chapter on alerting on SLOs explains burn-rate alerts if the interviewer asks how you would build the replacement.

Practice this one: /challenges/noisy-alert-fatigue

Communicating during an incident

Interviewers often ask, "Who do you tell, and what do you say?" A short, concrete answer works:

  • Declare it early. Downgrading an incident is cheaper than declaring one late.
  • Post updates on a schedule, even when nothing changed: what we know, what we are doing, when the next update comes.
  • Keep a timeline as you go. The postmortem depends on it.
  • Write for the reader. Support and customers need impact and the next update time, not your theory.

If the interview is live, show it: "At this point I would post: 'Search is slow for some users since 14:10. We suspect the 13:45 deploy and are rolling it back. Next update at 14:45.'"

Blameless postmortem answers

The behavioral side of the round usually asks you to describe an incident and what changed after it. The postmortem culture chapter is the reference: the goal is to find contributing causes and fix the system, not to find a person to blame.

What a strong answer includes:

  • Impact in concrete terms: who was affected, for how long, and how you know.
  • A timeline, including how it was detected.
  • Contributing causes, plural. "Why did the system let one mistake reach users?" is the question that leads to a fix.
  • Actions that can be checked later, each with an owner. "Be more careful" is not an action. "Add a migration check to the deploy pipeline that blocks forward-only migrations without a flag" is.

If the interviewer asks "whose fault was it?", answer about the system. That is the answer they are scoring.

FAQ

How do I answer "walk me through an outage" in an interview?

Use one real incident and tell it in order: how it was detected, how you scoped the impact, what you did to mitigate, how you found the cause, and what changed afterward. Say "I" for your own actions, keep it blameless, and name one decision you would make differently now.

What is the difference between an SRE troubleshooting interview and an incident response interview?

A troubleshooting round usually hands you a broken system, sometimes a live VM, and watches you debug it. An incident response round is often a discussion: a prompt like "the service is slow" plus follow-up questions on decisions such as rollback, failover and communication. Many loops mix the two, so ask the recruiter which format to expect.

Should I always roll back first?

No. Roll back when a recent change is the likely cause and rolling back is fast and safe. If the release ran a forward-only migration, or the change cannot be undone, fixing forward or another mitigation such as a feature flag may be safer. Say which fact drove your choice.

Do I need to know PromQL for an SRE interview?

Not every company uses Prometheus, but you should be able to explain how you would get p99 latency, error rate and traffic from whatever tool they use. Knowing one histogram_quantile query and why it uses rate() and le shows you have actually looked at tail latency.

Practice the decisions, not just the framework

Reading these answers helps. Defending them while someone argues back is a different skill, and it is the one the round tests. A general chatbot makes up an answer on the spot. DevOps Ledger grades you against an answer key written before you started. Each case above is one of those decisions. The free guides on Kubernetes observability and Linux performance cover the commands behind them.

Practice these scenarios

Free guides to read next

Practice defending your decisions

DevOps Ledger is a scenario-based DevOps and SRE interview practice platform: you pick a real production incident, defend your decision in a chat against a senior-engineer AI that pushes back, and get scored against an answer key written before you started.

One payment. 12 months of access. No subscription, nothing renews.

Get full access - $19