Services deep dive: systemd, init.d & service

In lesson 14 you started a web server with systemctl. Now let's look under the hood. What actually starts all those services when a Linux machine boots? Why do old tutorials say service httpd restart or /etc/init.d/apache2 start? And how do you turn your own script into a proper service?

PID 1: the first program

When Linux boots, the firmware starts the bootloader (GRUB), which loads the kernel. The kernel then starts exactly one program, with process ID 1. That program's job is to start everything else: networking, logging, SSH, cron, your web server. If PID 1 dies, the whole system goes down.

ps -p 1 -o pid,comm     # who is PID 1?
ls -l /usr/sbin/init    # "init" is really a link to systemd

    PID COMMAND
      1 systemd

Today both families use systemd as PID 1. It wasn't always that way:

You'll still meet the old ways constantly: in tutorials, on ancient servers, and in commands that were kept around so old scripts keep working. So you need to know both.

The old way: SysV init

Under SysV init, every service was a shell script in /etc/init.d/ that understood start, stop, restart and status. The system ran them in order at boot, based on the runlevel:

The old commands still exist, but on a systemd machine they all just pass your request on to systemctl:

service crond restart
chkconfig crond on
ls /etc/init.d
runlevel

Redirecting to /bin/systemctl restart crond.service
Note: Forwarding request to 'systemctl enable crond.service'.
functions  README
N 3

service cron restart
sudo /etc/init.d/cron restart
sudo update-rc.d cron enable
runlevel

Restarting cron (via systemctl): cron.service.

N 3

systemd: everything is a unit

systemd manages units. The most common kinds:

• .service: a program to run (sshd, cron, your web server)
• .target: a group of units, the modern version of runlevels
• .timer: a schedule, like cron (systemctl list-timers)
• .socket: start a service only when someone connects. Ubuntu 24.04 starts SSH this way, through ssh.socket.

Each unit is described by a small text file. There are two places they live, and one is more important:

systemctl cat sshd       # (Ubuntu: ssh) show the unit file, with its path

A unit file has three sections:

[Unit]                              ← what it is, and what it starts after
Description=OpenSSH server daemon
After=network.target

[Service]                           ← how to run it
ExecStart=/usr/sbin/sshd -D $OPTIONS   (the command — always a full path)
Restart=on-failure                     (if it crashes, start it again)

[Install]                           ← what "enable" should do
WantedBy=multi-user.target             (start it as part of normal boot)

What “enable” really does

systemctl enable doesn't start anything. It creates a symlink (a shortcut) inside the target named in WantedBy=. At boot, systemd starts everything linked in multi-user.target.wants:

ls -l /etc/systemd/system/multi-user.target.wants/

lrwxrwxrwx 1 root root 37 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx 1 root root 36 sshd.service -> /usr/lib/systemd/system/sshd.service

disable removes the link. And mask goes further: it links the unit to /dev/null, so nothing can start it until you unmask it. That's handy when you want to be sure something stays off.

systemctl power moves

systemctl list-units --type=service   # what's running
systemctl list-unit-files            # everything installed: enabled / disabled / masked / static
systemctl --failed                   # what crashed
systemctl cat NAME                   # read its unit file(s)
sudo systemctl edit NAME             # add an override ("drop-in") safely
sudo systemctl daemon-reload         # re-read unit files after YOU change them
sudo systemctl mask NAME             # make it impossible to start
systemctl get-default                # which target boots by default (runlevel)
systemd-analyze blame                # what slowed down booting?

Drop-ins are the clean way to change a package's service. sudo systemctl edit cron creates /etc/systemd/system/cron.service.d/override.conf. Only the lines you add there change, and package updates can't undo them. For example, to make a service restart even after a clean stop:

[Service]
Restart=always

Build your own service

Let's turn a script into a real service that starts at boot, restarts if it crashes, and logs to the journal. First the script. It loops forever and prints a line every minute:

#!/bin/bash
while true; do
  echo "still alive at $(date +%H:%M)"
  sleep 60
done

Then the unit file. Everything a service prints goes straight into the journal:

[Unit]
Description=Heartbeat demo service

[Service]
ExecStart=/usr/local/bin/heartbeat.sh
User=nobody
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo vi /usr/local/bin/heartbeat.sh
sudo chmod +x /usr/local/bin/heartbeat.sh
sudo vi /etc/systemd/system/heartbeat.service
sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat
journalctl -u heartbeat -f

sudo nano /usr/local/bin/heartbeat.sh
sudo chmod +x /usr/local/bin/heartbeat.sh
sudo nano /etc/systemd/system/heartbeat.service
sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat
journalctl -u heartbeat -f

• User=nobody: services should run as the least powerful user that works. If one gets hacked, the attacker only gets nobody's powers, not root's.
• Full paths in ExecStart=, just like cron.
• Forget chmod +x and the service fails with status=203/EXEC (“couldn't execute”).

printf '#!/bin/bash\nwhile true; do\n  echo "still alive at $(date +%%H:%%M)"\n  sleep 60\ndone\n' | sudo tee /usr/local/bin/heartbeat.sh
printf '[Unit]\nDescription=Heartbeat demo service\n\n[Service]\nExecStart=/usr/local/bin/heartbeat.sh\nUser=nobody\nRestart=on-failure\n\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/heartbeat.service

Service, cron job, or timer?

• Needs to run all the time (a web server, a bot) → a service.
• Needs to run on a schedule (a backup at 2am) → cron (Linux Sysadmin, lesson 5), or a systemd timer.
• Needs to run once at boot → a service with Type=oneshot.

Don't confuse it with sysctl!

systemctl controls services. sysctl (one letter away) changes kernel settings: networking, memory, security. The settings live as tiny files under /proc/sys:

sysctl net.ipv4.ip_forward                    # read (0 = off)
sudo sysctl -w net.ipv4.ip_forward=1          # change now (gone after reboot)
echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-router.conf
sudo sysctl --system                          # load all config files (permanent)

Both families read /etc/sysctl.d/*.conf at boot, so that's where your changes belong. Ubuntu's /etc/sysctl.conf is full of commented-out examples. Rocky's just explains the folders, and its package defaults live in /usr/lib/sysctl.d/.

Try it

Quick check