Kubernetes Troubleshooting Interview Questions: What to Run and What They Ask Next
- 13 min read
The short answer
Kubernetes troubleshooting interview questions hand you a broken workload and watch how you reason. Read the pod state, run kubectl describe and kubectl logs --previous before changing anything, and say what each result proves. Then expect pushback: the interviewer offers a quick fix like more memory or more replicas, and you explain why it fails.
A Kubernetes scenario based interview question is more than a prompt and a paragraph of answer. The interviewer describes a broken workload, waits for your first move, then pushes a shortcut that sounds reasonable: more memory, more replicas, retries. You have to explain why it fails and what you would do instead.
Below are eight common scenarios, each with the prompt, the commands to run first, the pushback line, and a weak answer next to a strong one. For the shape of the whole interview loop, see how to pass a DevOps interview.
The method interviewers want to hear
Every scenario below uses the same loop. Say it out loud so the interviewer can follow it:
- Read the state before you touch anything.
- Read why the control plane put it there: Events, exit codes, conditions.
- Name a hypothesis and the command that would confirm or kill it.
- Mitigate if users are affected, without destroying the evidence.
- Fix the desired state (the manifest, the app), not the symptom.
kubectl get pods -o wide
kubectl describe pod <pod> # Events at the bottom explain WHY
kubectl logs <pod> --previous # the container that already died
kubectl get events --sort-by=.lastTimestamp
Avoid kubectl delete pod as a first step. The Deployment recreates the same broken pod, and you lose the --previous logs and the old Events. The Kubernetes debugging guide goes through each failure state in more depth.
1. CrashLoopBackOff after a deploy
The prompt: "After a deploy, a pod is stuck in CrashLoopBackOff. A teammate says give it more CPU and memory and it will settle. How do you diagnose it?"
CrashLoopBackOff is not an error. It is a state: the container keeps exiting, and the kubelet waits longer between restarts (starting at 10 seconds and doubling up to 5 minutes). The job is to find out why it exits.
kubectl logs <pod> --previous
kubectl describe pod <pod> # Last State: Terminated, Reason, Exit Code
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
The exit code points you somewhere. 1 usually means the app failed on its own, often a missing env var or secret. 137 is SIGKILL: with Reason: OOMKilled it is the memory limit, without it something else sent SIGKILL, often the kubelet after a failed liveness probe when the app did not exit on SIGTERM in time. 127 is a command not found in the image. 0 means the process finished, and a Deployment restarts it anyway.
The pushback: "The app logs look totally clean, no errors. So the app is fine and Kubernetes is just being weird, no?"
Clean logs followed by a restart is a clue, not a dead end. If the Events show Liveness probe failed shortly after each start, the kubelet is killing a healthy app, and the fix is the probe, not the code.
| Weak answer | Strong answer |
|---|---|
| "Double the limits, or delete the pod so it comes back fresh." | "I read --previous and the exit reason first. This deploy added an env var, so a missing secret or config is my top suspect. More memory only fixes the OOMKilled case, and only after I have confirmed it." |
Practice this one: /challenges/k8s-crashloopbackoff
2. OOMKilled every few hours
The prompt: "A worker pod is OOMKilled every three to four hours under normal load. You can raise the limit and add replicas in minutes, or dig into a suspected leak. Production is degraded but not down. What do you do?"
This is a judgment question more than a command question. Confirm the OOM first:
kubectl describe pod <pod> | grep -A 5 "Last State" # Reason: OOMKilled, Exit Code: 137
kubectl top pod <pod> --containers # needs metrics-server
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources.limits.memory}'
Then look at memory over a few restart cycles. A flat line near the limit means the limit is too low for the working set. A sawtooth that climbs and resets on every restart points at a leak, and a higher limit only delays the next kill.
The pushback: "Bumping the memory limit is one line and makes the alert stop. Why spend a day on a profiler when the easy change clearly works?"
| Weak answer | Strong answer |
|---|---|
| "Raise it to 2Gi, add two replicas, look at it next sprint." | "Prod is degraded, so I would stabilize now, but I treat that as a stopgap with an owner and a date. Before I restart or rescale, I capture evidence, because a restart throws away the leaking process I need to profile." |
The full answer key covers what evidence to grab, which mitigations are acceptable here, and which popular workaround makes things worse.
Practice this one: /challenges/k8s-oom-scale-vs-fix
3. Pods stuck Pending on Insufficient cpu
The prompt: "You scaled a Deployment up and the new pods are Pending with 0/3 nodes are available: 3 Insufficient cpu. A teammate opens a PR that deletes the resources block so they schedule. Do you approve it?"
Pending is a placement problem. The container has not started yet, so logs are empty. Go to the scheduler's reason and the node's accounting:
kubectl describe pod <pod> # FailedScheduling event
kubectl describe nodes | grep -A 8 "Allocated resources" # requests already placed
kubectl top nodes # actual usage, for contrast
kubectl get resourcequota,limitrange -n <namespace>
The key fact: the scheduler places pods by their requests, not by live usage. Nodes can sit at 25% real CPU and still be full, because the requests already placed on them leave no room.
The pushback: "kubectl top nodes shows about 25 percent CPU. There is obviously room, so the scheduler is wrong and the requests are the bug. Why not remove them?"
| Weak answer | Strong answer |
|---|---|
| "Approve it. The nodes are idle, the requests are just paperwork." | "No. Without requests or limits the pods become BestEffort: the scheduler treats them as free, nodes get overcommitted, and they are the first evicted under node pressure. If the pods request 1000m and use 200m, I lower the request to measured usage plus headroom. If the requests are honest, I add capacity on purpose. I also check ResourceQuota and LimitRange." |
A common trap: remove only the requests and keep a limit, and Kubernetes sets the request equal to the limit, so the pod gets harder to schedule.
Practice this one: /challenges/k8s-pending-pods-remove-requests
4. Healthy pods restart under load
The prompt: "During peak traffic your pods enter a restart loop and latency gets worse, but the app is not crashing. kubectl describe shows liveness probe failures. A teammate wants to increase replicas. Do you agree?"
kubectl get pods -w # RESTARTS climbing at peak
kubectl describe pod <pod> | grep -iE "liveness|readiness|killing"
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].livenessProbe}'
kubectl top pod <pod> --containers # CPU near the limit?
A probe with timeoutSeconds: 1 and failureThreshold: 1 on a handler that also does real work will fail the moment CPU is saturated. The kubelet then restarts a warm pod, drops its in-flight requests, and pushes its traffic onto the pods that are left. That is a feedback loop.
The pushback: "Kubernetes restarted the pods, which means they were unhealthy. Are you sure this is not a real crash we are papering over?"
The evidence answers it: no panics in the logs, and Events saying the container failed its liveness probe and will be restarted. That is the kubelet's decision, not a crash.
| Weak answer | Strong answer |
|---|---|
| "Double the replicas, or remove the liveness probe." | "Liveness restarts, readiness sheds traffic. I would add a readiness probe so a saturated pod stops getting new requests, loosen the liveness timeout and failureThreshold, add a startupProbe for slow boots, and keep the liveness endpoint cheap and free of dependency checks. Then decide on scaling." |
Practice this one: /challenges/k8s-liveness-probe-restart-storm
5. Every rolling deploy causes 502s
The prompt: "Every rolling deploy causes a short burst of 502s at the load balancer. A teammate wants ingress retries and night-only deploys. Do you agree?"
Check whether the failing requests go to pods that are shutting down:
kubectl rollout status deploy/<name>
kubectl get pods -l app=<name> -o wide -w # watch for Terminating
kubectl get endpointslices -l kubernetes.io/service-name=<svc> -w
aws elbv2 describe-target-group-attributes --target-group-arn <arn> # deregistration delay
If the 502s match the IPs of Terminating pods, you have the shutdown race. When a pod is deleted, the kubelet starts shutdown at the same time as the control plane removes the pod from EndpointSlices. kube-proxy, the ingress controller, and the cloud load balancer each pick up that change on their own schedule. An app that exits as soon as it gets SIGTERM drops requests that are still being routed to it.
The pushback: "Kubernetes removes the pod from the Service before it kills it. By the time the app gets SIGTERM, no traffic can reach it, so the app cannot be the problem, can it?"
That is the misconception being tested: both happen in parallel.
| Weak answer | Strong answer |
|---|---|
| "Turn on retries and deploy at 2am." | "Retries on POSTs can create duplicate orders, and night deploys do nothing for scale-downs and node drains. I would make the app finish in-flight requests on SIGTERM, add a short preStop sleep to cover endpoint propagation (Kubernetes 1.30 and later have a native sleep action for preStop, so the image needs no sleep binary), set terminationGracePeriodSeconds above preStop plus drain time, add a readiness probe, use maxUnavailable 0 with maxSurge, and line up the load balancer deregistration delay." |
Practice this one: /challenges/k8s-rolling-deploy-502s
6. One call in twenty between services fails
The prompt: "Service A intermittently fails to reach Service B. Roughly one call in twenty times out. A teammate says add retries and move on. How do you diagnose it?"
The skill here is localizing the failure to a layer: name resolution, TCP connect, or waiting for the response. Use a debug container that has real tools:
kubectl debug -it <a-pod> --image=nicolaka/netshoot --target=<container>
# inside the debug shell:
cat /etc/resolv.conf # ndots:5 means extra lookups for short names
dig svc-b.<namespace>.svc.cluster.local
curl -s -o /dev/null -w '%{time_namelookup} %{time_connect} %{time_total}\n' http://svc-b:8080/healthz
kubectl get endpointslices -l kubernetes.io/service-name=svc-b -o wide
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100
kubectl get networkpolicy -A
Then call each backend pod IP directly in a loop. One bad pod still in the endpoint list fits "sometimes" well. If resolution time spikes, look at CoreDNS logs, load, and restarting replicas.
The pushback: "Most calls work, so the network is obviously fine and it is just the app being slow sometimes."
| Weak answer | Strong answer |
|---|---|
| "Wrap it in a retry, or restart Service B." | "One in twenty has a cause. I time each phase to see if it is DNS, connect, or response, then hit each endpoint directly. If one pod fails consistently, its readiness probe is too lax. Retries are fine as a resilience layer once I know why those calls fail." |
Practice this one: /challenges/k8s-intermittent-service-dns
7. p99 spikes while CPU looks fine
The prompt: "Your API's p99 jumps during bursts, but average CPU is only about 40% of the limit. A teammate says CPU is clearly fine, it must be the database, and wants to add replicas. Do you agree?"
CPU limits are enforced per CFS period, 100ms by default. A limit of 2 CPUs means 200ms of CPU time per 100ms period. If 32 threads run at once, they spend that in about 6ms of wall time, and the whole container waits for the rest of the period. The per-minute average still looks low. Measure it directly:
# inside the container, cgroup v2
kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat # nr_periods, nr_throttled, throttled_usec
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{container="api"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{container="api"}[5m]))
Put that ratio next to p99. If traces show the time spent inside the service, not in database spans, the database is cleared.
The pushback: "Removing CPU limits is how one pod eats a whole node on a shared cluster. Isn't it safer to keep the limit and scale out instead?"
| Weak answer | Strong answer |
|---|---|
| "CPU is at 40%, so it is the database. Add an HPA at 50% CPU." | "Averages hide throttling. I would check the throttled-periods ratio against p99 first. If it lines up, I set accurate requests, raise or remove the CPU limit for this service while keeping the memory limit, and make the runtime match its CPU, for example GOMAXPROCS on Go versions before 1.25. Removing the limit is only safe with honest requests, and I would check the namespace LimitRange and agree it with the cluster owners." |
Practice this one: /challenges/k8s-cpu-limits-throttling
8. An EKS pod needs AWS access
The prompt: "A pod in EKS needs to read from S3. A teammate says add the permission to the node instance role so it works today. Do you agree?"
This one is half troubleshooting, half security. Find out what identity the pod actually has:
kubectl get pod <pod> -o jsonpath='{.spec.serviceAccountName}'
kubectl get sa <sa> -n <namespace> -o yaml # eks.amazonaws.com/role-arn annotation (IRSA)
aws eks list-pod-identity-associations --cluster-name <cluster> # EKS Pod Identity
kubectl exec <pod> -- env | grep AWS_ # AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE
kubectl exec <pod> -- aws sts get-caller-identity # if the image has the AWS CLI
If get-caller-identity returns the node role, the pod is borrowing the node's credentials through instance metadata.
The pushback: "The node role already exists and the SDK picks it up automatically. Adding one S3 statement there is the quickest unblock, right?"
| Weak answer | Strong answer |
|---|---|
| "Add S3 read to the node role, tighten it later." | "Every pod on that node can reach the node role, so I would be granting S3 to all of them. I would give this workload its own role through IRSA or EKS Pod Identity, scoped to the bucket it needs, and stop pods from reaching the node role through instance metadata." |
The answer key also covers the other shortcuts candidates reach for when the interviewer says "it has to work today", and why each one is worse. The AWS IAM guide covers the role and policy basics.
Practice this one: /challenges/aws-eks-irsa-node-role
How to handle the pushback
Every pushback line above offers something cheaper than your answer: one line of YAML, more replicas, a retry. Some are right in a narrow case, such as more memory when the working set really exceeds the limit. Say when the shortcut would be correct, what evidence would show it, and why it does not fit here.
Two habits help:
- Tie every claim to a command. "The scheduler counts requests" lands harder with "and
kubectl describe nodesshows 95% already requested." - Change your mind only for new evidence. If the interviewer adds a fact, update out loud. If they only repeat the shortcut, hold your position politely.
Saying these answers while someone argues back is a separate skill from reading them. A general chatbot makes up an answer on the spot. DevOps Ledger grades you against an answer key written before you started. Each scenario above links to its challenge if you want to argue it out.
FAQ
What Kubernetes troubleshooting questions come up most in interviews?
Pod states are the usual starting point: CrashLoopBackOff, Pending, OOMKilled, ImagePullBackOff, and Running but not Ready. Senior rounds add judgment calls such as 502s during rollouts, intermittent service failures, CPU throttling, and pod identity on EKS.
What does exit code 137 mean in Kubernetes?
The container received SIGKILL (128 + 9). If kubectl describe pod shows Reason: OOMKilled, it went over its memory limit. Without that reason, something else killed it, often the kubelet after a failed liveness probe when the app did not exit on SIGTERM within the grace period. Exit code 143 is SIGTERM (128 + 15).
What is the difference between OOMKilled and an evicted pod?
OOMKilled means a container went over its own memory limit and the kernel killed it. The pod stays, and the container restarts according to its restart policy. Eviction means the kubelet removed the whole pod because the node itself was short on memory or disk. Pods using more than they requested are evicted first, which puts BestEffort pods at the front of the line, and the pod shows Status: Failed with Reason: Evicted.
How do I practice Kubernetes scenario based interview questions?
Break a local cluster on purpose (kind or minikube) and fix it while narrating each step. Then have someone push a bad shortcut at you and defend your answer. The Kubernetes fundamentals guide covers the background the scenarios assume.
Practice these scenarios
- Pod Stuck in CrashLoopBackOff After a Deploy
A new deploy leaves a pod in CrashLoopBackOff. A teammate wants to bump CPU and memory. Defend how you actually diagnose it.
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- Pods Stuck Pending on Insufficient CPU. Delete the Requests?
New pods sit Pending with Insufficient cpu and a teammate wants to delete the resource requests so they schedule. Defend the real fix.
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- Every Rolling Deploy Causes 502s. Add Ingress Retries?
Each rolling deploy causes a burst of 502s at the load balancer. A teammate wants ingress retries and night-only deploys. Defend the real fix.
- Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.
- p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.
- An EKS Pod Needs AWS Access. Attach It to the Node Role?
A pod needs to read S3, and a teammate wants to add the permission to the EKS node role to unblock it. Defend the least-privilege approach.
Free guides to read next
Practice defending your decisions
DevOps Ledger is a scenario-based DevOps and SRE interview practice platform: you pick a real production incident, defend your decision in a chat against a senior-engineer AI that pushes back, and get scored against an answer key written before you started.
One payment. 12 months of access. No subscription, nothing renews.
Get full access - $19