A 30-Second Blip Became a 2-Hour Outage. More Retries?
A brief downstream blip cascaded into a full-platform meltdown because every layer retried. A teammate wants to raise retry counts. Defend the real fix.
Answer key reviewed by Branko Petric
Read first: Kubernetes Observability: Logs, Metrics, and Monitoring
Your decision
A downstream payment service blipped for 30 seconds. Retries at the client, the API gateway, and two internal services piled on, and the whole platform stayed down for 2 hours after the payment service had already recovered. In the postmortem a teammate proposes raising retry counts from 3 to 5 so requests eventually succeed next time. What do you do, and why?
The situation
Last Tuesday a downstream payment service had a 30-second blip - a node restarted and health checks briefly failed. The platform did not recover in 30 seconds. Latency climbed, queues filled, and the entire platform was effectively down for 2 hours, long after the payment service itself was healthy again.
Context
The postmortem timeline shows retry logic at four places: the mobile client retries 3 times, the API gateway retries 3 times, and two internal services each retry 3 times with no backoff. During the incident the payment service saw more than 20x its normal request rate, almost all of it retries, and could not climb out. In the postmortem review a teammate proposes: "The requests failed because we gave up too early. Raise retry counts from 3 to 5 across the board so requests eventually succeed." You own the reliability recommendation.
Related challenges
- On-Call Is Drowning in Alerts. Silence Them?
A CPU alert pages constantly with no real impact. A teammate wants to silence it. Defend how you fix alerting properly.
- p99 Latency Spiked Across a Microservice Chain
p99 latency jumped across a request that touches six services. A teammate wants to scale up the slowest-looking one. Defend how you find the real culprit.
- The Queue Backlog Doubles Every Day. 10x the Consumers Tonight?
A consumer queue backlog is doubling daily and a teammate wants to 10x the consumer fleet tonight. Defend diagnosing the real bottleneck first, and why blind consumer scaling can make the outage worse.
- Chasing a Bug: Flip Production to DEBUG and Log Everything?
To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.