Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
Answer key reviewed by Branko Petric
Read first: Debugging Kubernetes: Diagnose Pods and Fix Failures
Your decision
A payments worker pod gets OOMKilled every 3 to 4 hours under normal load. You can raise the memory limit and add replicas in minutes, or dig into a suspected memory leak with unknown effort. Production is currently degraded but not down. What do you do, and why?
The situation
A Kubernetes payments worker (payments-worker) is being OOMKilled roughly every three to four hours under normal traffic. Each kill drops a few in-flight jobs, which retry, so customers see occasional delays but not hard failures.
Context
The pod has a 512Mi memory limit. Memory climbs steadily from startup until the kill, then resets on restart, a classic sawtooth. You have a heap profiler available but have not captured a profile yet. A colleague suggests "just bump it to 2Gi and add two more replicas, we can look at it next sprint."
Related challenges
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
- p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.
- Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.