Scenario Based DevOps Interview Questions, With Worked Answers

Branko Petric
Branko Petric

- 14 min read

The short answer

Scenario based DevOps interview questions hand you a broken system or a risky shortcut and ask what you would do. Strong answers start with a command that narrows the cause, name the cost, blast radius, security impact and reversibility of each option, and hold a position under pushback. Weak answers apply the shortcut or list tools.

Scenario questions describe a broken system, or a teammate proposing a shortcut, and ask what you would do. In the real round, the interviewer wants your first command, listens to why you chose it, then pushes back to see whether you fold.

This post covers twelve scenarios across Linux, Docker, CI/CD, Git and AWS. Each has the prompt, the first commands, a weak and a strong answer, and the pushback to expect. Kubernetes, Terraform and incident-command scenarios get separate posts. The classic full disk, CrashLoopBackOff, destructive Terraform plan and skip-the-tests prompts are already worked through in how to pass a DevOps interview.

What a scenario question is really testing

Almost every scenario contains a tempting shortcut: chmod 777, a bigger instance, a retry, a force-push. The shortcut usually works in the narrow sense, so that is not the point. The interviewer wants to see whether you check four things first:

Lens The question behind it Typical shortcut it catches
Cost Does this fix scale the bill instead of the cause? Add another NAT gateway, size up the box
Blast radius If this goes wrong, how much breaks, and for whom? Force-push main, run the app as root
Security Does this widen who or what can do damage? Mount the Docker socket, static keys in CI
Reversibility Can I undo this in a minute if I am wrong? Rewrite shared history, delete logs during an incident

Say the lens out loud: "The fast option here is irreversible, so I want to confirm the cause first." When pushback comes, notice which lens it pulls on. Usually it tests whether you will trade one away for speed.

Linux troubleshooting interview questions

"The deploy fails with Permission denied. A teammate wants to chmod -R 777 the app directory."

First commands:

systemctl show -p User -p Group myapp     # who the service runs as
namei -l /srv/app/data/uploads            # owner and mode of every path component
stat -c '%U:%G %a %n' /srv/app/data/uploads
sudo -u appuser touch /srv/app/data/uploads/.probe   # reproduce as the real user

Weak answer: "Run chmod 777 so permissions can never break a deploy again."

Strong answer: "Permission denied means one user lacks access to one path. If the directory is owned by root and the service runs as appuser, I fix that pair: chown -R appuser:appuser /srv/app/data, then 750 on directories and 640 on files, or 770/660 if a group must share it. Then I find why root owns it. Usually the deploy step writes files as root or the CI user, so I make it write as appuser or chown as its last step."

Pushback: "It is an internal box. Who are we protecting against?" Any process on the box, not just people with shell. One exploited dependency can then rewrite your code. And 777 hides the deploy bug, so it comes back.

Practice this one: the chmod 777 challenge

"Right after a deploy, nginx returns 502 for every request."

First commands:

tail -n 50 /var/log/nginx/error.log   # look for "connect() failed (111: Connection refused)"
ss -tlnp | grep 3000                  # is anything listening on the upstream port?
curl -sv http://127.0.0.1:3000/health # skip nginx entirely
journalctl -u myapp -n 100 --no-pager # why the new build did not start

Weak answer: "Restart nginx and tweak the proxy timeouts."

Strong answer: "A 502 means nginx got no valid response from the upstream. The proxy worked a minute ago, so the new build is the suspect. If curl to the local port fails too, nginx is just the messenger. The app log usually names it: a missing env var, a failed migration, a changed port. If I cannot fix it in a few minutes, I roll back, but I read the log first so we do not redeploy the same break."

Pushback: "The site is down, no time for logs." One log read is faster than three blind reloads.

Practice this one: the 502 after deploy challenge

"The service keeps getting OOM-killed. Add 16GB of swap and raise swappiness?"

First commands:

journalctl -k | grep -iE 'out of memory|killed process'   # what was killed, and its RSS
free -m
cat /proc/pressure/memory                                 # PSI: time stalled on memory
ps -eo pid,rss,cmd --sort=-rss | head
pidstat -r -p "$(pgrep -d, -f myservice)" 60              # RSS over time: leak or spike?

Weak answer: "Swap is extra memory, so more swap means fewer kills."

Strong answer: "For a latency-sensitive service, swap turns a fast kill and restart into a box that pages hot memory to disk and misses its SLOs while still alive. First I find out which problem this is. RSS that climbs steadily across hours is a leak, and a bigger box only delays it. A jump that tracks traffic is a spike. A flat line above RAM is a real working set. Then I fix the leak or size to the working set, set an explicit limit (heap size or cgroup memory.max), use oom_score_adj so the right process dies, and alert on memory pressure before the kill."

Pushback: "Every server has swap. Is swap bad?" No. Using it to hide memory pressure on a latency-sensitive service is.

Practice this one: the OOM and swap challenge

Docker scenario based interview questions

"Builds take 11 minutes and sometimes ship stale dependencies. Add --no-cache everywhere?"

First commands:

docker build --progress=plain . 2>&1 | grep -E '^#[0-9]+ (\[|CACHED)'   # which steps miss cache
docker history --no-trunc myapp:latest | head

Then read the Dockerfile. The usual culprit:

 FROM node:20
 WORKDIR /app
-COPY . .
-RUN npm ci
+COPY package.json package-lock.json ./
+RUN npm ci
+COPY . .

Weak answer: "Use --no-cache so every image is clean."

Strong answer: "Docker invalidates a layer and everything after it when its inputs change. With COPY . . above npm ci, every source edit reinstalls every dependency. That is where the minutes go. I copy the lockfile first, install, then copy the code. The stale dependency comes from a floating base tag and loose versions, so I pin the base image by digest and rely on the lockfile, which makes every cache hit a correct one. CI runners start cold, so I also add a cache backend."

docker buildx create --use --driver docker-container
docker buildx build \
  --cache-from type=registry,ref=registry.example.com/myapp:buildcache \
  --cache-to type=registry,ref=registry.example.com/myapp:buildcache,mode=max \
  -t registry.example.com/myapp:"$GIT_SHA" --push .

Pushback: "Just buy a bigger runner." It still rebuilds every layer from scratch, only slightly faster, for more money.

Practice this one: the Docker cache challenge

"The CI job needs to build images. Mount /var/run/docker.sock and run --privileged?"

First commands: here you show the risk rather than diagnose a fault. Anyone who can reach the socket can do this:

docker run --rm -it -v /:/host alpine chroot /host sh   # a root shell on the host

Weak answer: "Mount the socket, it is the simplest way to get Docker in the job."

Strong answer: "The socket controls a daemon that runs as root on the host, so mounting it gives every job root on the runner. This pipeline builds pull requests from outside contributors, so that is untrusted code with host root and access to other jobs' secrets. --privileged removes the capability and seccomp limits that keep a container contained. I would build with a daemonless builder like buildah or rootless BuildKit inside the job. If a daemon is truly needed, I would run an isolated docker-in-docker service per job, never the host socket."

Pushback: "Only our team's code runs here." Pull requests, dependencies and third-party actions run there too.

Practice this one: the Docker socket challenge

CI/CD pipeline failure interview questions

"Tests pass locally and fail in CI. Add a retry?"

First commands:

git clone --depth 1 "$REPO_URL" /tmp/ci-repro && cd /tmp/ci-repro   # no local caches
docker run --rm -v "$PWD":/src -w /src node:20 sh -c 'node --version && npm ci && npm test'
TZ=UTC npm test               # CI runners usually run in UTC
npx jest --randomize          # surface order-dependent tests (Jest 29.5+)

Weak answer: "CI is flaky. Retry until it is green."

Strong answer: "Only-in-CI is a signal, not noise. The usual causes are unpinned dependencies, a different OS or runtime, a missing env var, the timezone, or shared state like a temp directory. I reproduce the CI environment from a clean checkout, find the difference, and make the tests deterministic: pin versions, freeze time, isolate state, mock the network. A retry is fine for a genuinely external step, but it should be explicit and logged."

Pushback: "Re-running makes it pass, so it is random." Passing on re-run points to ordering, timing or shared state. That is a bug you can find, and blind retries are how a real one gets shipped.

Practice this one: the flaky CI challenge

"A third-party action you use as @v2 had its tags moved to a malicious commit for twelve hours. A teammate bumped the tag and wants to close the ticket."

First commands:

grep -rn 'uses: vendor/scanner@' .github/workflows/        # every reference, per repo
gh run list --workflow scan.yml --created "$WINDOW_START..$WINDOW_END" --limit 200
git ls-remote https://github.com/vendor/scanner 'refs/tags/v2*'   # resolve tags to SHAs

Weak answer: "Bump to the maintainer's new safe tag. The logs look normal."

Strong answer: "This is an incident in our environment, not a version bump. For each run in the window I list what secrets and permissions the job had, and rotate the long-lived ones (registry and npm tokens) now. Clean logs prove little: a payload can exfiltrate over the network or encode values past masking. GITHUB_TOKEN and OIDC sessions expire on their own, so for those I hunt for misuse instead: cloud audit events from the deploy role, unexpected image pushes, package versions, tags or workflow changes. I archive the logs as evidence before restricting public copies. Then I pin actions to commit SHAs, updated by Dependabot or Renovate, and enforce pinning by org policy."

- uses: vendor/scanner@3f1c9e7d2a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d # v2.4.1

Pushback: "The new tag is from the maintainer, it is safe." It is just as mutable as the old one. The March 2025 tj-actions/changed-files compromise (CVE-2025-30066) worked exactly this way.

Practice this one: the compromised action challenge

Git and security scenarios

"You pushed AWS keys to a public repo five minutes ago. Delete the file and force-push?"

First commands:

aws iam update-access-key --user-name app-user --access-key-id AKIAEXAMPLE --status Inactive
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLE \
  --max-results 50

Weak answer: "Delete the file, force-push, maybe make the repo private."

Strong answer: "The key is burned the moment it lands in a public repo, because scrapers watch public pushes. Rewriting history does not recall what was already cloned or forked. So I deactivate the key first, issue a replacement, and check CloudTrail for anything the key did. Then I clean history with git filter-repo or BFG. Prevention is secret scanning, push protection, and roles or OIDC instead of static keys."

Pushback: "Let us check the billing dashboard first to see if anyone used it." By the time abuse shows on the bill, it has happened. Deactivating is fast and reversible. For the CI half of this, see the OIDC vs static keys challenge.

Practice this one: the leaked credentials challenge

"main has a tangle of merge commits. A teammate wants to rebase it and force-push."

First commands:

git log --oneline --graph -20 origin/main
git revert -m 1 <bad-merge-sha>     # if something is actually wrong, fix it forward

Weak answer: "Force-push the clean version, it is a quick cleanup."

Strong answer: "Four people have clones and open branches based on the current main. Rewriting it changes the base under all of them. Their next pull conflicts, or someone merges stale work and the old commits come back. Ugly history is cosmetic. If a change is wrong, I revert it so the fix is a normal commit everyone pulls. To keep history clean from now on, I rebase feature branches before merge and turn on branch protection that blocks force-pushes to main."

Pushback: "Use --force-with-lease, that is the safe one." The lease only stops you overwriting commits you have not seen. It still rewrites everyone's base.

Practice this one: the force-push challenge

AWS scenario based interview questions

"NAT Gateway data processing is now the biggest line on the bill. Add another NAT gateway?"

First commands:

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-0abc123
aws ec2 create-vpc-endpoint --vpc-id vpc-0abc123 \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-0private1 rtb-0private2

Before creating anything, group Cost Explorer by usage type (look for NatGateway-Bytes) and query VPC flow logs for the top destinations.

Weak answer: "The gateway is saturated, add a second one."

Strong answer: "NAT bills per gigabyte processed. A second gateway adds an hourly charge and still bills every gigabyte. The traffic is S3 reads and ECR pulls, so it should not cross the NAT. A gateway endpoint for S3 has no hourly or data charge, and it also covers ECR layer downloads, because ECR serves layers from S3. For the ECR API and registry calls I add interface endpoints for ecr.api and ecr.dkr. Then I shrink images."

Pushback: "Interface endpoints cost money too." They do: an hourly charge per AZ plus a per-GB charge. Compare that to the NAT processing charge for the same traffic before adding each one. The S3 gateway endpoint is free, so start there.

Practice this one: the NAT Gateway cost challenge

"Lambda functions hit 'too many connections' on RDS during spikes. Raise max_connections?"

First commands:

SHOW max_connections;
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;

Then line up the CloudWatch metrics DatabaseConnections (AWS/RDS) and ConcurrentExecutions (AWS/Lambda) for the same spike.

Weak answer: "Raise max_connections in the parameter group."

Strong answer: "Each concurrent invocation runs in its own environment, and these functions open a fresh connection per request. Hundreds of concurrent invocations mean hundreds of connections at once. Each connection costs instance memory, so a higher limit trades the error for memory pressure, and the next spike hits the new ceiling. I move connection setup outside the handler so warm invocations reuse it, and put RDS Proxy in front to pool connections, watching for session pinning (SET statements and similar), which weakens multiplexing."

Pushback: "Just cap Lambda concurrency." That turns away user traffic. It is an emergency brake, not the fix.

Practice this one: the RDS connection storm challenge

"SQS workers sometimes process a message twice, causing double charges. Move to FIFO?"

First commands:

aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names VisibilityTimeout RedrivePolicy ApproximateNumberOfMessagesNotVisible

Weak answer: "Standard SQS is unreliable, switch everything to FIFO."

Strong answer: "Standard queues are at-least-once by design, so duplicates are expected. But here the visibility timeout is 30 seconds and jobs take a couple of minutes. The message reappears mid-job and a second worker picks it up. I raise the timeout above the maximum processing time, or extend it with ChangeMessageVisibility as a heartbeat. I delete only after success, and add a dead-letter queue with a maxReceiveCount. Then I make the side effect idempotent: a unique job ID recorded with the result, and an idempotency key sent to the payment provider."

Pushback: "FIFO has exactly-once processing, so why not?" FIFO deduplicates within a 5-minute window and has lower throughput limits, and it cannot make your charge or email call idempotent when a worker crashes halfway. You still need the dedup key.

Practice this one: the SQS duplicate processing challenge

How to answer scenario based DevOps interview questions for experienced roles

For senior roles, what you do with the results matters more than the commands. Interviewers listen for:

  • Ranked hypotheses. "Most likely the app crashed, less likely the port changed, so I check the listener first."
  • Mitigation before root cause when users are affected, said out loud.
  • The cost of your own fix. Interface endpoints have a bill. RDS Proxy has pinning. A DLQ needs someone to watch it.
  • The fix for the class. The deploy step that writes as root, the mutable tag, the static key. Fixing the one instance is the junior answer.
  • Holding a position for a reason. Update on a new fact, not on the same shortcut said louder.

Defending an answer while someone argues takes rehearsal. That is what DevOps Ledger is built for. A general chatbot makes up an answer on the spot. DevOps Ledger grades you against an answer key written before you started.

FAQ

What are scenario based DevOps interview questions?

They describe a broken system or a risky proposal and ask what you would do. You are scored on how you narrow the cause, which options you rule out, and how you handle pushback.

How should I structure an answer to a scenario question?

Restate the problem, name the first command and what its result would prove, give a short-term mitigation if users are affected, then the real fix and how you would stop it recurring. Name the trade-off of your chosen fix before the interviewer does.

Are scenario questions different for experienced engineers?

The prompts are often the same, but the bar moves: rank hypotheses, name the cost and blast radius of your own fix, and fix the class of problem, not just the instance.

What if I do not know the specific tool in the scenario?

Say so, then reason from first principles: which component is failing, what evidence would confirm it, and what the safest reversible step is. Bluffing a command usually gets caught on the follow-up.

How do I practice scenario based questions?

Out loud, against someone who argues back. Write your first three commands for a prompt, then have a peer push the shortcut at you.

Practice these scenarios

Free guides to read next

Practice defending your decisions

DevOps Ledger is a scenario-based DevOps and SRE interview practice platform: you pick a real production incident, defend your decision in a chat against a senior-engineer AI that pushes back, and get scored against an answer key written before you started.

One payment. 12 months of access. No subscription, nothing renews.

Get full access - $19