Troubleshooting a network
"The website is down." "I can't connect." "The internet is broken." You'll hear these a lot, and they almost never tell you where the problem is. Good troubleshooters don't guess. They use a method: test one layer at a time, from the bottom up, until they find the first one that fails. This lesson puts everything from this path together into that method.
You will learn
- The bottom-up ladder, with the command for each rung
- What each kind of failure points to
- Checking from both ends, and changing one thing at a time
- Fixing a real multi-layer outage on Rocky or Ubuntu
The ladder
Climb from the bottom. The first rung that fails is where the problem is, and everything above it is just a symptom.
| Rung | Question | Try | Good answer |
|---|---|---|---|
| 1. Link | Is the interface up, with a cable? | ip -br link, ethtool enp0s3 | UP, Link detected: yes |
| 2. Address | Do I have the right address? | ip -br addr | The address you expect, not 169.254… |
| 3. Local network | Can I reach my gateway? | ping -c 2 192.168.1.1 | Replies |
| 4. Routing | Can I get beyond it? | ip route, ping -c 2 8.8.8.8, tracepath -n | A default via line; replies |
| 5. Names | Does DNS work? | getent hosts NAME, dig NAME | An address |
| 6. Port | Does the service answer? | nc -zv HOST PORT, on the server ss -tlnp | Connected (not refused, not timed out) |
| 7. Application | Does it answer correctly? | curl -v URL | The status code and content you expect |
You don't always have to start at the very bottom: one ping to the target tests rungs 1–4 at once. If that works, jump straight to the port. If it fails, go down. That's divide and conquer.
What failures point to
| You see | Look at |
|---|---|
Network is unreachable | No route: often a missing default gateway (lesson 3) |
Destination Host Unreachable | Nothing answers on the local network: wrong address, or the device is off (lesson 1) |
Temporary failure in name resolution | DNS: which server is configured, and can you reach it? (lesson 4) |
| Timed out on a port | A firewall dropping packets, on the server or on the way (lesson 5) |
| No route to host on a port, while ping works | A firewall that rejects instead of dropping (Rocky's firewalld does this) |
| Connection refused | The machine is there, but nothing listens on that port and address: is the service running? Listening on 127.0.0.1 only? |
| A 4xx/5xx status | The network is fine: it's the web server or the app (lesson 6) |
| A certificate error | TLS (lesson 7) |
Habits of good troubleshooters
- Test from both ends. From the client, and on the server itself (
curl localhost). If it works on the server but not from outside, the problem is between them: firewall, binding or routing. - Change one thing at a time, and test after each change. Otherwise you never know what fixed it, or what you broke.
- Read the error word for word. "Refused" and "timed out" are different problems.
- Write down what you did. The next person (maybe you, at 3 a.m.) will thank you. That's the start of a postmortem (Linux SRE, lesson 4).
- Make fixes stick. An
ipcommand fixes things until the next reboot; the real fix goes in the config.
Where the network settings and web server's listening address are kept is Linux-specific:
| Fix | Rocky / RHEL | Ubuntu / Debian |
|---|---|---|
| Gateway and DNS, for good | sudo nmcli con mod enp0s3 ipv4.gateway … ipv4.dns … then sudo nmcli con up enp0s3 | routes: and nameservers: in /etc/netplan/*.yaml, then sudo netplan apply |
| Open the web port | sudo firewall-cmd --permanent --add-service=http + --reload | sudo ufw allow 'Apache' |
| Where Apache listens | Listen in /etc/httpd/conf/httpd.conf | Listen in /etc/apache2/ports.conf |
| A path view that keeps updating | sudo dnf install mtr, then mtr 8.8.8.8 | mtr 8.8.8.8 (mtr-tiny is preinstalled) |
Try it: "the website is down" 🧗
You're on your laptop. Nobody can open the club website at http://192.168.1.50, and the server can't download updates either. Climb the ladder, fix each rung, and prove it from both ends.
Quick check
1. curl localhost on the server works, but from the laptop nc -zv 192.168.1.50 80 times out. Where's the problem?
✓ It works inside, and the packets get no answer from outside: something drops them on the way in.
2. After opening the firewall, the laptop gets "connection refused" and ss -tlnp shows 127.0.0.1:80. What's wrong?
✓ Change Listen 127.0.0.1:80 to Listen 80 and restart Apache.
3. ping 8.8.8.8 says "Network is unreachable", but ping 192.168.1.1 works. Which rung?
✓ The local network is fine (the gateway answers). Check ip route for a default via line.