SRE incident and observability interview questions
SRE incident rounds start with a symptom: the service is slow, the queue keeps growing, the on-call is drowning in alerts. Interviewers want to see you protect users first, then narrow the cause with metrics and traces, and say what you would change afterward.
Each question below is the exact prompt from a DevOps Ledger challenge. Decide what you would do, then open the challenge to defend it.
16 scenarios: 1 beginner, 7 intermediate, 8 advanced. Answer keys reviewed by Branko Petric.
What interviewers probe
- Mitigating before you chase the root cause
- Latency percentiles, saturation and the other golden signals
- Retries, backoff and how load amplifies itself
- Paging on symptoms users feel, not on every cause
The questions
- 01intermediate12 min
On-Call Is Drowning in Alerts. Silence Them?
On-call keeps getting paged at 3am by an alert that fires whenever CPU goes over 80% for one minute, but nothing is actually wrong for users. A teammate says just silence that alert. What do you do, and why?
Practice this scenario - 02advanced15 min
p99 Latency Spiked Across a Microservice Chain
Your p99 latency jumped on a user request that flows through six microservices. Average latency looks fine. A teammate points at the service with the highest CPU and says scale that one up. How do you actually find the culprit?
Practice this scenario - 03advanced14 min
A 30-Second Blip Became a 2-Hour Outage. More Retries?
A downstream payment service blipped for 30 seconds. Retries at the client, the API gateway, and two internal services piled on, and the whole platform stayed down for 2 hours after the payment service had already recovered. In the postmortem a teammate proposes raising retry counts from 3 to 5 so requests eventually succeed next time. What do you do, and why?
Practice this scenario - 04intermediate12 min
The Queue Backlog Doubles Every Day. 10x the Consumers Tonight?
An order-processing queue backlog has grown from 20k to 160k messages over three days and is doubling daily. A teammate wants to 10x the consumer count tonight to burn it down before customers notice. Do you scale the consumers now, or do something else first? Defend your call.
Practice this scenario - 05advanced14 min
We're Missing a Dimension, So Add user_id as a Label?
Debugging is hard because metrics lack per-user detail, so a teammate wants to add user_id (and request_id) as Prometheus metric labels to slice everything by user. What do you do, and why?
Practice this scenario - 06intermediate12 min
Chasing a Bug: Flip Production to DEBUG and Log Everything?
An intermittent bug only shows up in production. A teammate wants to flip the entire app to DEBUG log level and log full request and response bodies until they catch it. Do you agree, and what do you do instead?
Practice this scenario - 07intermediate12 min
Pods Keep OOMKilling: Scale Up or Fix the Leak?
A payments worker pod gets OOMKilled every 3 to 4 hours under normal load. You can raise the memory limit and add replicas in minutes, or dig into a suspected memory leak with unknown effort. Production is currently degraded but not down. What do you do, and why?
Practice this scenario - 08beginner9 min
The t3 Instance Got Slow. Just Size It Up?
Your service runs fine for a while, then gets slow under sustained traffic on a t3.medium. CPU looks pinned near a flat ceiling. A teammate says just upgrade to a bigger t3. What is really happening, and what do you do?
Practice this scenario - 09intermediate13 min
A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A traffic spike took your service down even though horizontal autoscaling was enabled - it scaled too slowly to keep up. A teammate says just raise the max replicas way up. What is the real fix?
Practice this scenario - 10advanced15 min
Service A Intermittently Cannot Reach Service B
Service A intermittently fails to reach Service B inside the cluster - roughly one call in twenty times out, the rest are fine. A teammate says just add retries and move on. How do you actually diagnose this?
Practice this scenario - 11intermediate12 min
Every Deploy Hammers the Database for 10 Minutes. Bigger DB?
Every deploy flushes your Redis cache, and for the next 10 minutes the database runs at 95% CPU serving a flood of identical queries while the cache refills. Users see timeouts. A teammate proposes doubling the database instance size so it can absorb the refill. Do you upsize the database, or fix something else? Defend your call.
Practice this scenario - 12advanced13 min
The Box Keeps OOM-Killing. Just Add a Huge Swapfile?
A latency-sensitive service keeps getting OOM-killed on its VM, and a teammate says just add 16GB of swap and crank vm.swappiness so the kernel stops killing it. What do you do, and why?
Practice this scenario - 13advanced14 min
Under Load, Healthy Pods Keep Restarting. Add Replicas?
During peak traffic your pods enter a restart loop and latency gets worse, even though the app is not crashing. kubectl shows liveness probe failures and restarts. A teammate says just increase the replica count to spread the load. Do you agree, and what do you do instead?
Practice this scenario - 14intermediate13 min
After an Outage, Mandate Manual Approval on Every Deploy?
A bad deploy caused a two-hour outage. Leadership now wants a mandatory manual approval gate on every production deploy, where a lead signs off before anything ships. Do you agree, and what do you propose instead?
Practice this scenario - 15advanced15 min
It Is Just a Config File. Push It to Every Region at Once?
A rules file regenerated from a database every few minutes was hot-reloaded by every edge proxy worldwide, and a bad version crashed all of them at once. The team proposes a manual approval on every publish, raising the proxy's size limit, or keeping the global push but making it faster. Which do you pick, and what do you actually change?
Practice this scenario - 16advanced14 min
p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
Your pricing API shows p99 latency spikes during bursts, but its average CPU usage is only about 40 percent of the container's CPU limit. A teammate says CPU is clearly fine, so it must be the database, and wants to add replicas while they look. Do you agree, and how do you find and fix the cause?
Practice this scenario
Worked answers
- SRE Incident Response Interview Questions: The Decisions Interviewers Probe
SRE incident response interview questions with a worked 'the service is slow' walkthrough, PromQL, and the trade-offs: rollback, retries, failover, alerting.
Practice defending your decisions
DevOps Ledger is a scenario-based DevOps and SRE interview practice platform: you pick a real production incident, defend your decision in a chat against a senior-engineer AI that pushes back, and get scored against an answer key written before you started.
A general chatbot makes up an answer on the spot. DevOps Ledger grades you against an answer key written before you started.
One payment. 12 months of access. No subscription, nothing renews.
Get full access - $19