Chasing a Bug: Flip Production to DEBUG and Log Everything?
To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.
Observability
Securitythe decision you defend
An intermittent bug only shows up in production. A teammate wants to flip the entire app to DEBUG log level and log full request and response bodies until they catch it. Do you agree, and what do you do instead?
the situation
An intermittent bug corrupts a small fraction of orders, and it only reproduces in production. It never appears in staging. A teammate wants to set the application's global log level to DEBUG and add a middleware that logs full request and response bodies for every call until they manage to catch a failing case.
context
The service handles checkout, so requests include customer details and payment-related tokens. Logs are shipped to a hosted aggregator billed per gigabyte ingested and are retained for 30 days. The app already emits structured logs with a per-request correlation ID and has basic metrics and tracing wired up, though nobody has looked closely at the traces for the failing orders yet.
How this challenge works
Take a position on the decision above and defend it. A senior-engineer AI will push back over up to 5 rounds. When you are done, you are scored against a verified rubric so you can see exactly what a complete answer covers - these are learning prompts, not gotchas.