p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.
Answer key reviewed by Branko Petric
Read first: Kubernetes Observability: Logs, Metrics, and Monitoring
Your decision
Your pricing API shows p99 latency spikes during bursts, but its average CPU usage is only about 40 percent of the container's CPU limit. A teammate says CPU is clearly fine, so it must be the database, and wants to add replicas while they look. Do you agree, and how do you find and fix the cause?
The situation
A pricing API on Kubernetes serves synchronous requests for the checkout page. During traffic bursts its p99 latency jumps from tens of milliseconds to several hundred, while p50 barely moves. Dashboards show the pods' CPU usage averaging about 0.8 cores against a CPU limit of 2. The database team reports no change in query latency on their side, and in traces the slow requests show gaps inside the pricing service itself rather than long database spans.
Context
Each pod requests 1 CPU and has a CPU limit of 2 and a memory limit of 1Gi. The service is written in Go and built with a Go version older than 1.25, with no GOMAXPROCS setting, and it runs on nodes with 32 cores. Prometheus already scrapes the kubelet's cAdvisor metrics, but nobody has built a panel from the CFS metrics. The namespace has a LimitRange that sets a default CPU limit for containers that do not declare one. A teammate says the CPU graph proves CPU is fine, the database must be slow, and in the meantime the team should double the replicas and add an HPA at 50 percent CPU.
Related challenges
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
- Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.