Incidents, on-call & blameless postmortems

Your phone buzzes: SiteDown. Every SRE, sysadmin and DevOps engineer gets this moment eventually, and what you do in the next ten minutes matters more than any tool. This lesson covers how teams handle incidents calmly: who does what, how to communicate, why you roll back first and debug later, and how to write a blameless postmortem so the same outage never happens twice. Then you'll handle a real one.

What counts as an incident?

An incident is any unplanned problem that hurts (or is about to hurt) users and needs a coordinated response. Teams sort them by severity, so everyone knows how loud to be:

Every company sets its own definitions. What matters is that they're written down before the incident, so nobody argues about them in the middle of one.

The lifecycle

1. Detect: monitoring pages you (ideally before users notice). Time to detect is MTTD.
2. Respond: acknowledge the page so others know someone has it, open an incident channel, and assess the severity.
3. Mitigate: stop the pain. Roll back the last change, fail over, or turn off the broken feature. You don't need to understand the bug to undo a change.
4. Resolve: confirm users are OK, then say so publicly. Time to restore is MTTR.
5. Learn: write the postmortem, fix the root cause, and track the action items until they're done.

Roles: nobody does everything

For big incidents, one person fixing, coordinating and answering “is it fixed yet?” messages quickly gets overwhelmed. Teams split the work. (The idea comes from emergency services' Incident Command System.)

Small incidents, like the one you're about to handle, are usually one person doing all four. Keep a timeline anyway: date +%H:%M and a notes file are enough.

On-call that doesn't burn people out

• Rotation: people take turns, often a week each, with a proper handoff (“here's what's flaky right now”). Teams across time zones do follow-the-sun, so nobody gets paged at 3 am.
• Escalation: if the primary doesn't acknowledge within a few minutes, the tool (PagerDuty, Opsgenie, Grafana OnCall…) pages the secondary, then a manager.
• Runbooks: every alert links to a page that says what it means and what to try first. You'll use one in the practice.
• Keep it humane: count the pages. If on-call keeps getting woken up, fixing that becomes the team's top reliability project.

Blameless postmortems

After the incident, write down what happened and what will change. The golden rule is that it's blameless: it asks what allowed this to happen?, never who messed up?

• If people get blamed, they hide mistakes, and the next outage takes longer to understand.
• “Human error” is never a root cause. A person made a typo? Everyone makes typos. The real question is why a typo could take the site down with nothing to catch it.
• Keep asking “why?” (the 5 whys) until you reach something you can fix: Site down → Apache didn't start → config typo → nothing tested the config before the restart → the edit was made by hand, directly on production.
• Every postmortem ends with action items, each with an owner and a due date, tracked like any other work. A postmortem without follow-through is just a story.

Summary · Impact · Timeline (UTC) · Root cause · What went well
What went wrong · Where we got lucky · Action items (owner, due date)

Tools you'll use in an incident

systemctl status httpd
sudo apachectl configtest
journalctl -u httpd --since "1 hour ago"
sudo journalctl -t sudo --since "1 hour ago"
sudo grep COMMAND /var/log/secure
ls -lt /etc/httpd/conf.d
diff good.conf current.conf

systemctl status apache2
sudo apache2ctl configtest
journalctl -u apache2 --since "1 hour ago"
sudo journalctl -t sudo --since "1 hour ago"
sudo grep COMMAND /var/log/auth.log
ls -lt /etc/apache2/sites-available
diff good.conf current.conf

Practice: you're on call 🚨

The pager just went off. Everything you need is in ~/incident. Follow the runbook, get the site back, keep people updated, then write the postmortem. The timeline in the log says who made the change, but remember: blameless.

Quick check