We're Missing a Dimension, So Add user_id as a Label?
Debugging lacked per-user detail, so a teammate wants user_id and request_id as Prometheus labels. Prometheus is already under memory pressure. Defend the call.
Observabilitythe decision you defend
Debugging is hard because metrics lack per-user detail, so a teammate wants to add user_id (and request_id) as Prometheus metric labels to slice everything by user. What do you do, and why?
the situation
During a recent incident nobody on the team could break down error rate by customer, so triage dragged while people grepped logs by hand. In the retro, a teammate proposes adding high-cardinality labels like user_id and request_id to the app's Prometheus metrics so that every dashboard can be sliced by user going forward.
context
The Prometheus server is already showing memory pressure and has had a couple of OOM restarts under load. Metrics today carry bounded labels like route, status, and region. Logs and distributed tracing exist but are underused. A teammate in chat says: just add user_id and request_id as labels, it's one small change and it would have saved us hours last week.
How this challenge works
Take a position on the decision above and defend it. A senior-engineer AI will push back over up to 4 rounds. When you are done, you are scored against a verified rubric so you can see exactly what a complete answer covers - these are learning prompts, not gotchas.