Secrets & config: the twelve-factor way
Every app needs settings: where the database is, which port to use, whether debug mode is on. Some of those settings are secrets: passwords, API keys, tokens. Get this wrong and your database password ends up in git, in a log file, or on a public website, and bots find leaked keys within minutes. This lesson is about keeping config out of your code and secrets out of everything that gets shared.
You will learn
- The twelve-factor rule: config lives in the environment, not in the code or the image
- How secrets leak: git history, command lines, logs, world-readable files
.envfiles,.gitignore,chmod 600and${VAR:?}in compose- systemd
EnvironmentFile=:/etc/sysconfigon Rocky,/etc/defaulton Ubuntu - Container secrets, secret managers, and rotation: what to do after a leak
The twelve-factor rule
The Twelve-Factor App is a short, famous list of habits for building apps that deploy well. Factor III, Config, says: store config in the environment. The same image or code should run on your laptop, in staging and in production, with only the environment variables changing:
DB_HOST=localhost DB_PASSWORD=devpass # your laptop DB_HOST=db.staging DB_PASSWORD=(a staging secret) # staging DB_HOST=db.prod DB_PASSWORD=(a real secret) # production
A good test: could you make the repository public right now without leaking anything? If not, config is hiding in your code.
How secrets leak
| Leak | Why it's bad | Do this instead |
|---|---|---|
| Committed to git | Deleting it in a new commit doesn't help: git history keeps it forever, in every clone | Keep secrets in an ignored file or a secret manager, and scan commits |
On a command line (--password=…) | Every user on the server can see other users' command lines with ps aux. It also lands in shell history. | Read secrets from a file or an environment variable |
| In logs | “Connecting with password hunter2…” ends up in journald, log files and log collectors | Never log secrets. Mask them (****). |
| World-readable file | Mode 644 means every local user can read it | chmod 600 (or 640 with a group), owned by the service's user |
| Baked into a container image | Anyone who can pull the image can read every layer, even ones where you “deleted” the file | Pass secrets in when the container runs, never at build time |
.env files with compose
# .env (next to compose.yaml; NEVER committed) DB_PASSWORD=7f3a9c1e5b2d8f4a6c0e9b1d3f5a7c9e # compose.yaml (committed, and safe to share) environment: REDIS_PASSWORD: ${DB_PASSWORD:?set it in .env}
echo "DB_PASSWORD=$(openssl rand -hex 16)" > .env # a strong random password chmod 600 .env # only you can read it echo .env >> .gitignore # git will never pick it up git status # .env must NOT appear
${DB_PASSWORD:?message} makes compose stop with an error if the variable is missing, instead of quietly using an empty password. Many projects also commit a .env.example with the variable names and fake values, so a new teammate knows what to fill in.
Services: EnvironmentFile=
For a systemd service (like ourapp in Linux Sysadmin), put the variables in a root-only file and point the unit at it. The two families even agree on the idea, but not on the folder:
sudo tee /etc/sysconfig/ourapp <<'EOF'
DB_PASSWORD=7f3a9c1e5b2d8f4a…
EOF
sudo chmod 600 /etc/sysconfig/ourapp
# in the unit (or a drop-in):
[Service]
EnvironmentFile=/etc/sysconfig/ourappsudo tee /etc/default/ourapp <<'EOF'
DB_PASSWORD=7f3a9c1e5b2d8f4a…
EOF
sudo chmod 600 /etc/default/ourapp
# in the unit (or a drop-in):
[Service]
EnvironmentFile=/etc/default/ourappsystemd reads the file as root before dropping privileges, so the file can stay 600 root while the app runs as its own user. Newer systemd also has LoadCredential=, which hands the service a secret as a file instead of a variable. Environment variables can leak into crash reports and child processes, and credentials can't.
Beyond files
- Container secrets:
podman secret create db_pw file(or Docker/composesecrets:) mounts the secret as a file at/run/secrets/db_pwinside the container, never in the image and never ininspectoutput. - Secret managers: HashiCorp Vault or its open-source fork OpenBao, cloud ones (AWS Secrets Manager, Azure Key Vault, Google Secret Manager), and encrypted files kept in git (SOPS, ansible-vault). One place to store secrets, control who can read them, see who did, and rotate them.
- CI secrets: GitHub Actions and GitLab store secrets encrypted and hand them to jobs as
${{ secrets.NAME }}, masked in the logs. - Scanners: gitleaks and trufflehog search your history for keys, and GitHub scans public repos and warns key providers automatically.
After a leak: rotate, don't just delete
Once a secret has been in git, a log or a chat, assume someone has it. Removing it from the file, or even rewriting git history, doesn't un-leak it: clones, forks and backups still have it. The fix is rotation: make a new secret, deploy it everywhere, and make sure the old one no longer works. Then check the logs for anyone who used the old one.
Practice: un-leak a password 🔐
A teammate protected the counter app's Redis with a password, and committed it. Find it, prove why that's bad, move it out of git, rotate it, and prove the old one is dead.
Quick check
1. You committed a password, noticed, and made a new commit that removes it. Are you safe?
✓ git log -p shows everything that was ever committed.
2. Why is mysql --password=hunter2 on the command line risky on a shared server?
✓ That's why redis-cli warns about -a, and why tools read passwords from files or prompts.
3. Where does a systemd service's secret belong on Rocky? And on Ubuntu?
✓ Root-only file, read by systemd before the service drops privileges.
4. What does ${DB_PASSWORD:?set it in .env} do in compose.yaml?
✓ Failing loudly beats silently running with an empty password.