On-Call Is Drowning in Alerts. Silence Them?
A CPU alert pages constantly with no real impact. A teammate wants to silence it. Defend how you fix alerting properly.
Answer key reviewed by Branko Petric
Read first: Kubernetes Observability: Logs, Metrics, and Monitoring
Your decision
On-call keeps getting paged at 3am by an alert that fires whenever CPU goes over 80% for one minute, but nothing is actually wrong for users. A teammate says just silence that alert. What do you do, and why?
The situation
Your on-call engineers are exhausted. The pager fires several times a night for an alert that triggers whenever CPU usage exceeds 80% for one minute. Every time, they check and find no user-facing impact.
Context
The service handles the spikes fine; latency and error rates stay normal even when CPU is briefly high. The alert was added long ago and pages directly. A teammate, tired of the 3am pages, wants to silence the alert entirely.
Related challenges
- p99 Latency Spiked Across a Microservice Chain
p99 latency jumped across a request that touches six services. A teammate wants to scale up the slowest-looking one. Defend how you find the real culprit.
- A 30-Second Blip Became a 2-Hour Outage. More Retries?
A brief downstream blip cascaded into a full-platform meltdown because every layer retried. A teammate wants to raise retry counts. Defend the real fix.
- The Queue Backlog Doubles Every Day. 10x the Consumers Tonight?
A consumer queue backlog is doubling daily and a teammate wants to 10x the consumer fleet tonight. Defend diagnosing the real bottleneck first, and why blind consumer scaling can make the outage worse.
- We're Missing a Dimension, So Add user_id as a Label?
Debugging lacked per-user detail, so a teammate wants user_id and request_id as Prometheus labels. Prometheus is already under memory pressure. Defend the call.