Security basics · Lesson 7 · 30 min

Did anyone change anything?

Logs tell you what happened, if nobody cleaned them up. Files tell you what's different. If you saved a fingerprint of every important file while the server was healthy, you can always ask later: what was added, what was removed, what changed? That's file integrity checking, and it catches things logs miss.

You will learn

What a checksum is, and why one changed letter changes all of it
Asking the package manager whether its files are still untouched
AIDE: a fingerprint database for the whole system, on both families
Telling a harmless change from a dangerous one

Checksums: a fingerprint for a file

A checksum (or hash) like SHA-256 turns a file into a short string. Same file, same string, every time. Change one letter and the string is completely different. You met them in Linux Sysadmin, lesson 2 and Linux Sysadmin, lesson 8:

sha256sum /usr/bin/passwd
sha256sum /usr/bin/passwd /usr/bin/sudo > ~/good.sha256    # save them
sha256sum -c ~/good.sha256                                 # check later: OK or FAILED

That works for a few files. For a whole server you want a tool that does it for thousands of files at once.

Ask the package manager

Your package manager remembers the fingerprint of every file it installed. It can tell you which ones no longer match:

Task	Rocky / RHEL	Ubuntu / Debian
Check one package	`rpm -V procps-ng`	`sudo dpkg --verify procps`
Check every package	`sudo rpm -Va`	`sudo dpkg --verify`
A changed program looks like	`S.5....T. /usr/bin/ps`	`??5?????? /usr/bin/ps`
A changed config file looks like	`S.5....T. c /etc/ssh/sshd_config`	`??5?????? c /etc/ssh/sshd_config`
Put a package's files back	`sudo dnf reinstall procps-ng`	`sudo apt install --reinstall procps`

The letters say what's different: S size, 5 contents, T time, M permissions, U/G owner. A c marks a config file, and config files are supposed to change when you edit them. A program that changed is a different story: nobody edits /usr/bin/ps by hand.

AIDE: fingerprint the whole system

The package manager only knows its own files. AIDE (Advanced Intrusion Detection Environment) fingerprints everything you tell it to, including files no package owns, like /etc/passwd or your own scripts. You make the database while the server is known-good, and compare against it later.

Task	Rocky / RHEL	Ubuntu / Debian
Install	`sudo dnf install aide`	`sudo apt install aide`
Config file	`/etc/aide.conf`	`/etc/aide/aide.conf`
Make the first database	`sudo aide --init`, then `sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz`	`sudo aideinit`
Compare with the database	`sudo aide --check`	`sudo aide --config /etc/aide/aide.conf --check`
Accept the changes as the new normal	`sudo aide --update`, then `sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz`	`sudo aide --config /etc/aide/aide.conf --update`, then `sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db`

Keep a copy of the database somewhere else

Someone who gets root can change files and the database that would catch them. Copy the database to another machine (or a USB stick) after every update. The same goes for the tools: rkhunter and chkrootkit look for known rootkits, but they also run on the server they're checking.

Harmless or dangerous?

Integrity tools tell you what changed, not why. You decide:

A config file you or a teammate edited, with a note saying why: fine. Accept it with --update.
A new account with UID 0: that's a second root, under another name. Check with awk -F: '$3 == 0' /etc/passwd. Only root should be there.
A system program that no longer matches its package: someone with root replaced it, often to hide what they're doing. A swapped ps can hide a process from you.

Root was taken? Rebuild.

A UID 0 account and a swapped program both mean someone had root. In the lab below you'll contain it and put things back so you can see the tools go green again. On a real server, the safe answer is the one from lesson 5: take it off the network, install fresh, restore your data from backups, and find out how they got in.

Try it: Monday's check 🧬

Last week you made an AIDE database of the club server while it was healthy. Today you compare. Sam (a teammate) says they changed "one SSH setting" on the weekend. Is that all that changed?

Quick check

1. rpm -Va shows S.5....T. c /etc/ssh/sshd_config. Should you panic?

2. Why is an account with UID 0 so serious?

3. When should you run aide --update and accept a new database?

Finished the missions and the quiz? Mark it done to track your progress.