Disk Is at 100% and the App Is Down. rm -rf?
A server is out of disk and the app is down. A teammate wants to rm -rf the logs to free space fast. Defend what you actually do first.
Answer key reviewed by Branko Petric
Read first: Linux for DevOps: The Fundamentals
Your decision
A web server is throwing 'No space left on device' and the app is down. df shows / at 100%. A teammate says just run rm -rf on /var/log to free space fast. What do you do, and why?
The situation
It is the middle of a workday. The main web server starts returning 500s and the application log shows repeated 'No space left on device' errors. You SSH in and df -h shows / at 100%.
Context
It is a single VM running the app behind nginx. Application and access logs are written to /var/log, and no log rotation was ever configured. A teammate in chat says: just rm -rf /var/log and restart the service, that always works.
Related challenges
- The Fix Is One Line in /etc. Just SSH In and Edit It?
Prod is degraded by one wrong sysctl value in /etc on an Ansible-managed fleet. A teammate wants to hand-edit each box now and codify it later. Defend the call.
- Deploy Fails with Permission Denied. chmod 777 Everything?
A deploy fails with permission errors and a teammate wants to chmod -R 777 the app directory so it never happens again. Defend fixing the actual ownership mismatch instead.
- The Box Keeps OOM-Killing. Just Add a Huge Swapfile?
A latency-sensitive service keeps getting OOM-killed and a teammate wants to add 16GB of swap and crank swappiness. Defend what you actually do instead.
- 3 AM Outage: Roll Back or Fix Forward?
A deploy 20 minutes ago correlates with a spike in 500s. You are on call. Defend whether to roll back or fix forward.