Pods Stuck Pending on Insufficient CPU. Delete the Requests?

New pods sit Pending with Insufficient cpu and a teammate wants to delete the resource requests so they schedule. Defend the real fix.

beginner8 min4 roundsEasy mode availableKubernetes

Answer key reviewed by Branko Petric

Read first: Debugging Kubernetes: Diagnose Pods and Fix Failures

Your decision

You scaled a Deployment up and the new pods are stuck Pending with "0/3 nodes are available: 3 Insufficient cpu". A teammate opens a PR that deletes the resources block so the scheduler stops complaining. Do you approve it, and what do you do instead?

The situation

A checkout service is scaled from 6 to 10 replicas ahead of a sale. Six pods are Running, four sit in Pending. kubectl describe pod on one of them shows a FailedScheduling event: "0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod."

Context

The cluster has three worker nodes with roughly 3.9 CPU allocatable each. Every checkout pod requests 1000m CPU and 512Mi memory, with limits of 2 CPU and 1Gi, values copied from an old template. kubectl describe node shows CPU requests at around 95 percent of allocatable on each node, while kubectl top nodes shows about 25 percent actual CPU use and kubectl top pods shows each checkout pod using around 200m. The cluster autoscaler is installed, but the node group is already at its configured maximum of 3. A teammate opens a PR that deletes the whole resources block from the Deployment so the pending pods schedule before the sale.

More: Kubernetes scenario interview questions