Security basics · Lesson 5 · 30 min

Spot trouble in the logs

Every server on the internet gets knocked on all day long. Most of it is bots trying passwords and giving up. Your job isn't to panic about the noise. It's to notice the one line that's different: the login that worked. This lesson is a real investigation, the way admins do it, from the first clue to the locked door.

You will learn

  • Where each family keeps its login log
  • Turning thousands of lines into answers with grep, sort and uniq -c
  • last, lastb, who and w: who came, who failed, who's here now
  • What to do, in what order, when someone got in

Where the login log lives

You met logs in Linux Sysadmin, lesson 4. For security, one log matters most: the one where SSH and sudo write every login and every refusal.

WhatRocky / RHELUbuntu / Debian
Login & sudo log file/var/log/secure/var/log/auth.log
Same, from the journaljournalctl -u sshdjournalctl -u ssh
Logins (and who's still on)last same on both
Failed loginssudo lastb same on both
Who is logged in right nowwho or w same on both

The lines to know

Log line saysIt means
Invalid user admin from 203.0.113.45Someone tried a user that doesn't exist here. Pure noise, usually.
Failed password for root from …A real account, a wrong password. Lots of these from one address = someone guessing.
Accepted password for … / Accepted publickey for …A login that worked. Every one of these should be someone you know, from somewhere you expect.
user NOT in sudoersSomeone tried to become root and was refused. Your own users do this by mistake sometimes. Intruders do it on purpose.

From noise to answers

A busy log has thousands of lines. Three tools from Linux Sysadmin, lesson 3 turn it into a short list:

sudo grep -c "Failed password" /var/log/secure            # how many?
sudo grep "Failed password" /var/log/secure | grep -o "from [0-9.]*" | sort | uniq -c | sort -rn
                                                            # who, and how often?
sudo grep "Accepted" /var/log/secure                        # did anyone get IN?

(On Ubuntu, use /var/log/auth.log instead.) The last one is the most important command in this lesson. Many failures followed by an Accepted from the same address means a password was guessed.

Someone got in: what now?

Order matters. Close the door before you chase them out, or they'll just walk back in:

  1. Look first, change second. Note what you see (last, who, the log lines). In a real incident, copy the logs somewhere safe before touching anything.
  2. Lock the account: sudo usermod -L NAME and sudo usermod -s /sbin/nologin NAME.
  3. Throw them out: sudo pkill -KILL -u NAME ends every process of that user, including their login.
  4. Remove what they left behind, especially SSH keys in ~/.ssh/authorized_keys. Locking a password does not stop key logins on most setups, which is exactly why attackers add a key.
  5. Close the hole: turn off password logins (lesson 2) so guessing can't work again.
When to rebuild instead

Here the logs show sudo refused the intruder, so they never became root, and cleaning up is reasonable. If someone did get root, you can't trust anything on that server anymore: they could have changed any program or log. The safe answer then is a fresh install and your backups (Linux Sysadmin, lesson 8).

Let a robot watch the log: fail2ban

fail2ban reads the login log for you and blocks an address in the firewall after a few failures. Install it with sudo dnf install epel-release then sudo dnf install fail2ban on Rocky, or sudo apt install fail2ban on Ubuntu. Turn on the [sshd] jail in /etc/fail2ban/jail.local, then check with sudo fail2ban-client status sshd. It's a good extra layer, but keys-only SSH is what really stops guessing.

Try it: the account nobody remembered 🕵️

Monday morning. The club server feels fine, but it's your job to check. Read the evidence, find out what happened, and lock it down.

Quick check

1. Your log has 4,000 Failed password lines from all over the world, and no Accepted lines except your own. What does it mean?

2. You find an intruder logged in as test. What do you do first?

3. You locked test with usermod -L. Why remove its authorized_keys too?

Finished the missions and the quiz? Mark it done to track your progress.