It Is Just a Config File. Push It to Every Region at Once?
A generated rules file reached every edge proxy in every region within seconds, crashed all of them at once, and now the team wants a fix. Defend how config should ship.
CI/CD
ObservabilityAnswer key reviewed by Branko Petric
Read first: Continuous Deployment with GitHub Actions
Your decision
A rules file regenerated from a database every few minutes was hot-reloaded by every edge proxy worldwide, and a bad version crashed all of them at once. The team proposes a manual approval on every publish, raising the proxy's size limit, or keeping the global push but making it faster. Which do you pick, and what do you actually change?
The situation
Your edge network runs proxies in every region. They filter and route traffic using a rules file that a pipeline regenerates every few minutes from a database query and publishes to a central store. Every proxy watches the store and hot-reloads the new file within seconds. Last week a permissions change on the database made the query return each row twice. The file doubled in size, crossed a hard-coded limit in the proxy's loader, and every proxy crashed on load at roughly the same moment. Traffic failed globally until someone found the file and pushed an older copy by hand.
Context
Application code for the proxy goes through CI, a canary on a few hosts and a staged rollout across regions over several hours. The rules file does not, because the team has always treated it as data rather than code. The generator does no checks on its output beyond confirming the query succeeded. A normal regeneration changes only a handful of entries, and the store keeps previous versions of the file. When the proxy's loader fails on a new file, the process exits; it has no fallback to the version it was already running. It took the on-call team a while to connect the crashes to the file, because no code had been deployed. In the retro three proposals are on the table: require a manual approval before each publish, raise the proxy's limit so a file of that size would load, or keep the global push but make propagation faster so a fix lands sooner next time. You are asked which one to adopt.