Pods Stuck Pending on Insufficient CPU. Delete the Requests?
New pods sit Pending with Insufficient cpu and a teammate wants to delete the resource requests so they schedule. Defend the real fix.
Answer key reviewed by Branko Petric
Read first: Debugging Kubernetes: Diagnose Pods and Fix Failures
Your decision
You scaled a Deployment up and the new pods are stuck Pending with "0/3 nodes are available: 3 Insufficient cpu". A teammate opens a PR that deletes the resources block so the scheduler stops complaining. Do you approve it, and what do you do instead?
The situation
A checkout service is scaled from 6 to 10 replicas ahead of a sale. Six pods are Running, four sit in Pending. kubectl describe pod on one of them shows a FailedScheduling event: "0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod."
Context
The cluster has three worker nodes with roughly 3.9 CPU allocatable each. Every checkout pod requests 1000m CPU and 512Mi memory, with limits of 2 CPU and 1Gi, values copied from an old template. kubectl describe node shows CPU requests at around 95 percent of allocatable on each node, while kubectl top nodes shows about 25 percent actual CPU use and kubectl top pods shows each checkout pod using around 200m. The cluster autoscaler is installed, but the node group is already at its configured maximum of 3. A teammate opens a PR that deletes the whole resources block from the Deployment so the pending pods schedule before the sale.
Related challenges
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- Every Rolling Deploy Causes 502s. Add Ingress Retries?
Each rolling deploy causes a burst of 502s at the load balancer. A teammate wants ingress retries and night-only deploys. Defend the real fix.