The Queue Backlog Doubles Every Day. 10x the Consumers Tonight?

A consumer queue backlog is doubling daily and a teammate wants to 10x the consumer fleet tonight. Defend diagnosing the real bottleneck first, and why blind consumer scaling can make the outage worse.

intermediate12 min4 roundsObservability

Answer key reviewed by Branko Petric

Read first: Kubernetes Observability: Logs, Metrics, and Monitoring

Your decision

An order-processing queue backlog has grown from 20k to 160k messages over three days and is doubling daily. A teammate wants to 10x the consumer count tonight to burn it down before customers notice. Do you scale the consumers now, or do something else first? Defend your call.

The situation

Your order-processing pipeline pushes each new order onto a queue, and a fleet of 20 consumers picks them up to charge payment, reserve inventory, and send confirmation emails. Three days ago the backlog started growing: 20k, then 40k, then 80k, and this morning it crossed 160k messages. Inbound order volume is normal for the season. Oldest-message age is now over six hours and support is seeing "where is my confirmation" tickets.

Context

The consumers write to the same Postgres instance as the main app and call a payment provider that rate-limits per account. Nobody has yet looked at per-consumer processing time, redelivery counts, or downstream latency; there is a dead-letter queue configured but its alarm was never wired up. A recent deploy touched the inventory-reservation code path. In the incident channel a teammate writes: "consumers are stateless, let us just set the autoscaling group to 200 tonight and the backlog is gone by morning."

More: SRE incident and observability interview questions