Logs & troubleshooting

Servers keep a diary. Every login, crash, error and cron job gets written down. When something breaks, a good admin doesn't guess. They read the logs. It's the closest thing Linux has to a detective's notebook. 🕵️

Two ways to read the diary

Modern Linux keeps logs in two places at once:

• The journal, collected by systemd and read with journalctl. It works the same on both families.
• Plain text files in /var/log, written by a program called rsyslog. Here the file names differ:

A log line always tells you when, which computer, which program [process ID], and what happened:

Sep 27 18:12:37 rocky sshd[2403]: Failed password for invalid user admin from 203.0.113.45 port 41822 ssh2
  when            host  program[PID]  message

Reading files: tail, grep, less

sudo tail -n 20 /var/log/secure
sudo grep "Failed password" /var/log/secure
sudo tail -f /var/log/messages

tail -n 20 /var/log/auth.log
grep "Failed password" /var/log/auth.log
tail -f /var/log/syslog

tail -f (follow) keeps watching and prints new lines the moment they appear. Admins leave it running in one window while testing in another. Ctrl+C stops it.

Detective trick: count the attackers

Any server on the internet gets constant password-guessing from bots. Chain tools with pipes to find out who's knocking:

sudo grep "Failed password" /var/log/secure | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c

grep "Failed password" /var/log/auth.log | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c

Read it left to right: find the failed lines → pull out just the IP addresses (grep -o prints only the match) → sort them so duplicates sit together → count them (uniq -c).

journalctl: one tool, both families

journalctl -n 20                 # the last 20 lines of everything
journalctl -u sshd -n 20         # just one service (the unit)
journalctl -p err                # only errors and worse
journalctl --since "1 hour ago"
journalctl -f                    # follow live, like tail -f
journalctl -xeu httpd            # jump to the end of one service, with explanations

Remember that service names differ: -u sshd vs -u ssh, -u crond vs -u cron, -u httpd vs -u apache2.

dmesg: the kernel's diary

The kernel (the engine from Linux Basics, lesson 1) keeps its own log of hardware events: disks being found, USB sticks plugged in, network cards coming up, and serious errors like a disk failing or memory running out.

dmesg | tail
dmesg -T | grep -i error    # -T = human-readable times

sudo dmesg | tail
sudo dmesg -T | grep -i error

The same messages are also in the journal: journalctl -k (kernel) works on both. You'll use dmesg to spot a brand-new disk in lesson 7.

The troubleshooting routine

When a service misbehaves, pros walk through the same steps every time:

1. Is it running? systemctl status NAME. The last few log lines appear right at the bottom.
2. What did it say? journalctl -xeu NAME for the full story.
3. Read the error slowly. It usually names the file, the line number, and the problem.
4. Fix one thing, then test the config before restarting. Apache has httpd -t (Rocky) and apache2ctl configtest (Ubuntu).
5. Restart and verify: sudo systemctl restart NAME, then test it for real (e.g. curl localhost).

Try it: the website is down! 🚨

Someone edited the web server's config last night and now the site won't start. Use the logs to find out why, then fix it. (Editing a file opens a mini editor: type, then Ctrl+O to save and Ctrl+X to exit, or use the buttons.)

Quick check