p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?

A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.

advanced14 min5 roundsKubernetesObservability

Answer key reviewed by Branko Petric

Read first: Kubernetes Observability: Logs, Metrics, and Monitoring

Your decision

Your pricing API shows p99 latency spikes during bursts, but its average CPU usage is only about 40 percent of the container's CPU limit. A teammate says CPU is clearly fine, so it must be the database, and wants to add replicas while they look. Do you agree, and how do you find and fix the cause?

The situation

A pricing API on Kubernetes serves synchronous requests for the checkout page. During traffic bursts its p99 latency jumps from tens of milliseconds to several hundred, while p50 barely moves. Dashboards show the pods' CPU usage averaging about 0.8 cores against a CPU limit of 2. The database team reports no change in query latency on their side, and in traces the slow requests show gaps inside the pricing service itself rather than long database spans.

Context

Each pod requests 1 CPU and has a CPU limit of 2 and a memory limit of 1Gi. The service is written in Go and built with a Go version older than 1.25, with no GOMAXPROCS setting, and it runs on nodes with 32 cores. Prometheus already scrapes the kubelet's cAdvisor metrics, but nobody has built a panel from the CFS metrics. The namespace has a LimitRange that sets a default CPU limit for containers that do not declare one. A teammate says the CPU graph proves CPU is fine, the database must be slow, and in the meantime the team should double the replicas and add an HPA at 50 percent CPU.

More: Kubernetes scenario interview questions