Final challenge: the 3 a.m. outage
3:07 a.m. Your phone buzzes: the club's website is down. You log in and the server is a mess. Several things have gone wrong at once, and they're hiding behind each other. Fix everything, and make sure it stays fixed after a reboot.
No step-by-step instructions. The objectives say what must be true when you're done. Start where a professional would: look before you touch (systemctl status, the logs, df -h, top, ip route). Everything you need is in Linux Sysadmin; the cheat sheet and search (/) are allowed, and hints are below the terminal.
The brief
- The website is back. The web server is running again, and the thing that stopped it is fixed properly, not worked around.
- The server can reach the internet, by name (
ping google.comworks), and the fix is saved so it survives a reboot. - There's room on the disk again: the root filesystem is below 90% full.
- The intruder is gone. Something is using all the CPU. Stop it, remove it, and make sure it can't start itself again.
- Never again. A nightly cron job makes a
tarbackup of/var/www, so next time you can restore the site in a minute.
Stuck? Hints
Open only as many as you need.
1. The website
Ask systemd why it failed: systemctl status and journalctl -xeu the web service. The error names a file and a line. Check your fix with httpd -t (Rocky) or apachectl configtest (Ubuntu) before restarting.
Still stuck: lesson 4.
2. The internet
Climb the ladder: address (ip -br a), gateway (ip route), then DNS (cat /etc/resolv.conf, resolvectl status). Two things are wrong. Fix them in the saved config (nmcli con mod on Rocky, the Netplan file on Ubuntu), then apply it.
Still stuck: lesson 1.
3. The disk
df -h says how full; du and find / -size +1G say what. Emptying a log that a program still has open is safer with truncate than rm.
Still stuck: lesson 6.
4. The intruder
top shows the process. Before you kill it, find out how it starts: look at scheduled jobs (crontab -l). Remove the schedule, stop the process, delete its files, then timewarp 10m to prove it stays gone.
5. The backup
A cron line has five time fields and a command. 0 2 * * * means 2:00 every night. The command could be tar czf ~/www-backup.tar.gz /var/www. crontab -e edits your jobs.