Linux Sysadmin · Lesson 9 · 30 min

Deploying your company's app

Not all software comes from dnf or apt. Sooner or later, your own company's developers hand you a program: “Here's ourapp, can you put it on the server?” Doing that well uses almost everything in this course: folders, users, permissions, links, systemd and logs. This lesson is the capstone.

You will learn

  • Where a company's own app goes: /opt/cht/ourapp, /etc/opt, /var/opt
  • Service accounts, and who should own which files
  • The releases/ + current symlink trick for painless upgrades and rollbacks
  • Running it with systemd, and the SELinux label trap that only bites on Rocky

Why not just put it anywhere?

You could leave ourapp in your home folder and start it by hand. Then you go on vacation, the server reboots, and nobody knows where the app lives, how to start it, or which files are safe to back up. A tidy layout means any admin, including future you, can find everything without asking.

The /opt family

The filesystem standard splits an add-on app into three folders, by what kind of file it is. This is what makes backups and upgrades easy:

PathWhat goes thereOwner & mode
/opt/cht/ourapp/The program itself. It never changes except during an upgrade.root:root, 755
/etc/opt/cht/ourapp/Its settings, including secretsroot:ourapp, 640
/var/opt/cht/ourapp/Data the app writes: its database, uploads, cacheourapp:ourapp, 750
/etc/systemd/system/ourapp.serviceHow systemd starts it (Linux Basics, lesson 15)root:root, 644
/usr/local/bin/ourappA link to the program, so people can just type ourappsymlink
journalctl -u ourappIts logs. Modern apps just print, and systemd collects the output.(the journal)

Now the backup plan writes itself: back up /etc/opt and /var/opt (lesson 8), and the program can always be reinstalled from its release file. Some vendors cram everything, settings and logs included, into /opt/vendor/app. You'll see both styles in the wild. The split version is easier to manage.

A service account

The app should run as its own user, never as root and never as you. If someone finds a bug in ourapp and takes it over, they only get ourapp's powers: they can't read other people's files, change the program, or touch the rest of the system. That's the whole permissions lesson (Linux Basics, lesson 10) paying off.

Rocky / RHEL
sudo useradd --system --shell /sbin/nologin ourapp
getent passwd ourapp
ourapp:x:999:999::/home/ourapp:/sbin/nologin

System accounts count down from 999 and get no home folder. nologin means nobody can log in as it.

Ubuntu / Debian
sudo adduser --system --group ourapp
getent passwd ourapp
ourapp:x:107:111::/nonexistent:/usr/sbin/nologin

adduser --system counts up from 100 and uses the home /nonexistent. --group gives it its own group. Note the path: /usr/sbin/nologin here, /sbin/nologin on Rocky.

Here's a trick used everywhere from school servers to huge tech companies. Keep every version in its own folder, and point a link called current at the one you want:

/opt/cht/ourapp/ ├── releases │ ├── ourapp-1.4.2 ← last week's version, kept just in case │ └── ourapp-1.5.0 ← the new one └── current -> releases/ourapp-1.5.0 ← systemd always starts current/bin/ourapp
Same on both
# upgrade: unpack the new version next to the old one, switch the link, restart
sudo tar -xzf ourapp-1.5.0.tar.gz -C /opt/cht/ourapp/releases/
sudo ln -sfn /opt/cht/ourapp/releases/ourapp-1.5.0 /opt/cht/ourapp/current
sudo systemctl restart ourapp

# something broke? roll back in two seconds
sudo ln -sfn /opt/cht/ourapp/releases/ourapp-1.4.2 /opt/cht/ourapp/current
sudo systemctl restart ourapp
The -n in ln -sfn matters

If current already points to a folder, plain ln -sf NEW current doesn't replace the link. It quietly creates a new link inside the old version's folder, and you're still running the old version. -n means “treat current as a link, not as the folder it points to.” Also note that switching the link doesn't change the running app. It keeps running the old version until you restart it.

Settings and secrets

Same on both
sudo cp /opt/cht/ourapp/current/share/ourapp.conf.example /etc/opt/cht/ourapp/ourapp.conf
sudo sed -i "s/CHANGE-ME/$(openssl rand -hex 32)/" /etc/opt/cht/ourapp/ourapp.conf
sudo chown root:ourapp /etc/opt/cht/ourapp/ourapp.conf
sudo chmod 640 /etc/opt/cht/ourapp/ourapp.conf     # root writes, ourapp reads, others: nothing

The $(openssl rand -hex 32) part makes a long random secret and drops it straight into the file, so you never have to see or copy it (lesson 2). Mode 640 with the ourapp group lets the app read its secret while ordinary users can't.

Before starting the service, test the app as its own user. Many real apps have a “check my settings” option like this one:

Rocky / RHEL
sudo -u ourapp /opt/cht/ourapp/current/bin/ourapp --check-config

The full path is needed: sudo's secure_path on Rocky leaves out /usr/local/bin, so sudo -u ourapp ourapp says “command not found” (Linux Basics, lesson 7).

Ubuntu / Debian
sudo -u ourapp ourapp --check-config

Run it with systemd

The developers included a unit file. It's worth reading every line before you install it:

[Unit]
Description=ourapp - CHT ticket tracker
After=network-online.target

[Service]
User=ourapp                     # never root
Group=ourapp
ExecStart=/opt/cht/ourapp/current/bin/ourapp --config /etc/opt/cht/ourapp/ourapp.conf
WorkingDirectory=/var/opt/cht/ourapp
Restart=on-failure              # crash? systemd starts it again

[Install]
WantedBy=multi-user.target
Same on both
sudo cp /opt/cht/ourapp/current/share/ourapp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now ourapp
systemctl status ourapp
curl localhost:8080

The trap: SELinux labels on Rocky

Say you unpacked the app in your home folder first and then sudo mv'd it into /opt. That's a very common way to do it. On Ubuntu, it works. On Rocky, the service fails with status=203/EXEC … Permission denied, even though ls -l shows perfectly good permissions. What's going on?

Rocky / RHEL
ls -Z /opt/cht/ourapp/current/bin
unconfined_u:object_r:user_home_t:s0 ourapp
sudo ausearch -m avc -ts recent
sudo restorecon -Rv /opt/cht
sudo systemctl restart ourapp

SELinux gives every file a label based on where it belongs. mv keeps the old one (“this came from a home folder”), and SELinux won't let system services run home-folder programs. ausearch shows what SELinux blocked, and restorecon resets labels to match where the files are now. Or avoid the problem: unpack straight into place with sudo tar -C, or use cp, because new files get the right label automatically.

Ubuntu / Debian
ls -Z /opt/cht/ourapp/current/bin
? ourapp

Ubuntu uses AppArmor instead. It protects specific programs using rules about paths (in /etc/apparmor.d/) and doesn't label files. ourapp has no AppArmor profile, so nothing stops it. The ? means “no label.”

Red flag: “just run setenforce 0”

Search for this error and you'll find plenty of answers (from people and from AI) saying to switch SELinux off. That's like fixing a squeaky door by removing it. The real fix is one restorecon command, and SELinux keeps protecting the whole server. See AI Basics, lesson 1.

Letting other computers in

ourapp listens on port 8080. curl localhost:8080 works on the server itself, but other machines also have to get through the firewall:

Rocky / RHEL
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload
Ubuntu / Debian
sudo ufw allow 8080/tcp
The next level: packages

Once an app is installed on more than a couple of servers, companies usually turn it into a real .rpm or .deb package and run their own repository. Then installing and upgrading is just dnf install ourapp or apt install ourapp, and rpm -qf / dpkg -S know which files belong to it (Linux Basics, lesson 11). Containers (Docker, Podman) are another popular option. The ideas here (a separate user, config apart from code, data in its own folder, logs to the journal) carry over to all of them.

Try it: deploy ourapp 🚀

Priya from the dev team left two release files and a note in your home folder. Install version 1.4.2 properly, then upgrade to 1.5.0.

Quick check

1. Where should ourapp's settings file go?

2. Why should the program files in /opt/cht/ourapp be owned by root and not by ourapp?

3. You switched current to version 1.5.0, but curl localhost:8080 still says 1.4.2. Why?

4. On Rocky, the service fails with 203/EXEC Permission denied, but ls -l shows -rwxr-xr-x root root. What should you check next?

Finished the missions and the quiz? Mark it done to track your progress.