p99 Latency Spiked Across a Microservice Chain
p99 latency jumped across a request that touches six services. A teammate wants to scale up the slowest-looking one. Defend how you find the real culprit.
Answer key reviewed by Branko Petric
Read first: Kubernetes Observability: Logs, Metrics, and Monitoring
Your decision
Your p99 latency jumped on a user request that flows through six microservices. Average latency looks fine. A teammate points at the service with the highest CPU and says scale that one up. How do you actually find the culprit?
The situation
Users report the app feeling slow. Your dashboards show p99 latency has jumped sharply on a key request that passes through about six microservices, while average latency looks normal.
Context
You have per-service CPU and memory metrics and request logs, and distributed tracing is available. One service in the chain is running noticeably hotter on CPU than the others. A teammate wants to scale that hottest service up and see if it helps.
Related challenges
- Chasing a Bug: Flip Production to DEBUG and Log Everything?
To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.
- On-Call Is Drowning in Alerts. Silence Them?
A CPU alert pages constantly with no real impact. A teammate wants to silence it. Defend how you fix alerting properly.
- A 30-Second Blip Became a 2-Hour Outage. More Retries?
A brief downstream blip cascaded into a full-platform meltdown because every layer retried. A teammate wants to raise retry counts. Defend the real fix.
- We're Missing a Dimension, So Add user_id as a Label?
Debugging lacked per-user detail, so a teammate wants user_id and request_id as Prometheus labels. Prometheus is already under memory pressure. Defend the call.