Linux DevOps · Lesson 5 · 35 min

Secrets & config: the twelve-factor way

Every app needs settings: where the database is, which port to use, whether debug mode is on. Some of those settings are secrets: passwords, API keys, tokens. Get this wrong and your database password ends up in git, in a log file, or on a public website, and bots find leaked keys within minutes. This lesson is about keeping config out of your code and secrets out of everything that gets shared.

You will learn

  • The twelve-factor rule: config lives in the environment, not in the code or the image
  • How secrets leak: git history, command lines, logs, world-readable files
  • .env files, .gitignore, chmod 600 and ${VAR:?} in compose
  • systemd EnvironmentFile=: /etc/sysconfig on Rocky, /etc/default on Ubuntu
  • Container secrets, secret managers, and rotation: what to do after a leak

The twelve-factor rule

The Twelve-Factor App is a short, famous list of habits for building apps that deploy well. Factor III, Config, says: store config in the environment. The same image or code should run on your laptop, in staging and in production, with only the environment variables changing:

DB_HOST=localhost   DB_PASSWORD=devpass            # your laptop
DB_HOST=db.staging  DB_PASSWORD=(a staging secret)   # staging
DB_HOST=db.prod     DB_PASSWORD=(a real secret)      # production

A good test: could you make the repository public right now without leaking anything? If not, config is hiding in your code.

How secrets leak

LeakWhy it's badDo this instead
Committed to gitDeleting it in a new commit doesn't help: git history keeps it forever, in every cloneKeep secrets in an ignored file or a secret manager, and scan commits
On a command line (--password=…)Every user on the server can see other users' command lines with ps aux. It also lands in shell history.Read secrets from a file or an environment variable
In logs“Connecting with password hunter2…” ends up in journald, log files and log collectorsNever log secrets. Mask them (****).
World-readable fileMode 644 means every local user can read itchmod 600 (or 640 with a group), owned by the service's user
Baked into a container imageAnyone who can pull the image can read every layer, even ones where you “deleted” the filePass secrets in when the container runs, never at build time

.env files with compose

# .env  (next to compose.yaml; NEVER committed)
DB_PASSWORD=7f3a9c1e5b2d8f4a6c0e9b1d3f5a7c9e

# compose.yaml  (committed, and safe to share)
    environment:
      REDIS_PASSWORD: ${DB_PASSWORD:?set it in .env}
echo "DB_PASSWORD=$(openssl rand -hex 16)" > .env   # a strong random password
chmod 600 .env                                       # only you can read it
echo .env >> .gitignore                              # git will never pick it up
git status                                           # .env must NOT appear

${DB_PASSWORD:?message} makes compose stop with an error if the variable is missing, instead of quietly using an empty password. Many projects also commit a .env.example with the variable names and fake values, so a new teammate knows what to fill in.

Services: EnvironmentFile=

For a systemd service (like ourapp in Linux Sysadmin), put the variables in a root-only file and point the unit at it. The two families even agree on the idea, but not on the folder:

Rocky / RHEL
sudo tee /etc/sysconfig/ourapp <<'EOF'
DB_PASSWORD=7f3a9c1e5b2d8f4a…
EOF
sudo chmod 600 /etc/sysconfig/ourapp

# in the unit (or a drop-in):
[Service]
EnvironmentFile=/etc/sysconfig/ourapp
Ubuntu / Debian
sudo tee /etc/default/ourapp <<'EOF'
DB_PASSWORD=7f3a9c1e5b2d8f4a…
EOF
sudo chmod 600 /etc/default/ourapp

# in the unit (or a drop-in):
[Service]
EnvironmentFile=/etc/default/ourapp

systemd reads the file as root before dropping privileges, so the file can stay 600 root while the app runs as its own user. Newer systemd also has LoadCredential=, which hands the service a secret as a file instead of a variable. Environment variables can leak into crash reports and child processes, and credentials can't.

Beyond files

After a leak: rotate, don't just delete

A leaked secret is a burned secret

Once a secret has been in git, a log or a chat, assume someone has it. Removing it from the file, or even rewriting git history, doesn't un-leak it: clones, forks and backups still have it. The fix is rotation: make a new secret, deploy it everywhere, and make sure the old one no longer works. Then check the logs for anyone who used the old one.

Practice: un-leak a password 🔐

A teammate protected the counter app's Redis with a password, and committed it. Find it, prove why that's bad, move it out of git, rotate it, and prove the old one is dead.

Quick check

1. You committed a password, noticed, and made a new commit that removes it. Are you safe?

2. Why is mysql --password=hunter2 on the command line risky on a shared server?

3. Where does a systemd service's secret belong on Rocky? And on Ubuntu?

4. What does ${DB_PASSWORD:?set it in .env} do in compose.yaml?

Finished the missions and the quiz? Mark it done to track your progress.