Deploys Are SSH + git pull at 5pm Friday. Why Change It?
The team ships by SSHing into prod and running git pull, often at 5pm Friday. A teammate says it has always worked. Defend what a real deploy needs without cargo-culting.
Answer key reviewed by Branko Petric
Read first: Continuous Deployment with GitHub Actions
Your decision
Your team deploys by SSHing into the prod server and running git pull, and the current release is planned for 5pm on Friday. A teammate says this has worked for two years, so why change anything? What do you argue, and why?
The situation
Your team runs a monolith on two VMs behind a load balancer. The release process is: SSH into each box, run git pull, sometimes run npm install if someone remembers dependencies changed, and systemctl restart the service. The next release is scheduled for 5pm this Friday because "that is when the sprint ends."
Context
There is a CI server that runs tests on pull requests, but nothing after merge; the prod boxes just pull main. Last month a deploy half-worked because one VM had a local hotfix edit that conflicted with the pull, and nobody noticed for a day. When you raise the process in retro, a teammate says: we have deployed this way for two years and it has always worked, why add ceremony?
Related challenges
- You Pushed AWS Keys to a Public Repo
Live cloud keys just landed in a public GitHub repo. A teammate says delete the file and force-push. Defend what you do first.
- History Is Messy, So Force-Push a Clean main?
A bad merge left main tangled with merge and revert commits. A teammate wants to rebase locally and force-push a clean history. Defend what you actually do.
- After an Outage, Mandate Manual Approval on Every Deploy?
After a bad deploy caused an outage, leadership wants a mandatory human sign-off on every production release. Defend a better way to prevent the next one.
- It Is Just a Config File. Push It to Every Region at Once?
A generated rules file reached every edge proxy in every region within seconds, crashed all of them at once, and now the team wants a fix. Defend how config should ship.