Chasing a Bug: Flip Production to DEBUG and Log Everything?

To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.

intermediate12 min5 rounds

Observability

Security

Your decision

An intermittent bug only shows up in production. A teammate wants to flip the entire app to DEBUG log level and log full request and response bodies until they catch it. Do you agree, and what do you do instead?

The situation

An intermittent bug corrupts a small fraction of orders, and it only reproduces in production. It never appears in staging. A teammate wants to set the application's global log level to DEBUG and add a middleware that logs full request and response bodies for every call until they manage to catch a failing case.

Context

The service handles checkout, so requests include customer details and payment-related tokens. Logs are shipped to a hosted aggregator billed per gigabyte ingested and are retained for 30 days. The app already emits structured logs with a per-request correlation ID and has basic metrics and tracing wired up, though nobody has looked closely at the traces for the failing orders yet.