Incident Runbooks People Actually Use During an Outage

The 3 a.m. test

A runbook has exactly one reader that matters: an engineer who was asleep twenty minutes ago, is looking at this system for the first time, and has a VP asking for updates. If your runbook opens with two paragraphs of architecture history, it has already failed that reader.

Structure that works under stress

1. Impact check (30 seconds). One query or one dashboard link that answers "are users actually affected, and how badly?" Everything else waits.
2. Mitigate first. The fastest action that stops user pain — rollback, failover, feature-flag off, scale up. Root cause comes later. Write the exact command, not "roll back the service".
3. Escalation line. Who to page if step 2 didn't work, with the actual pager alias. "Contact the platform team" is not an escalation path at 3 a.m.
4. Diagnosis tree. Only now: if X, check Y. Keep each branch to a command plus what healthy output looks like.

Write commands, show healthy output

The difference between a wiki page and a runbook is that a runbook is executable:

> kubectl -n payments rollout undo deploy/checkout
# healthy: "deployment.apps/checkout rolled back"
# then watch error rate fall within ~2 min:
> curl -s $DASH/api/error-rate | jq .last5m   # healthy: < 0.5

Showing the healthy output matters as much as the command — the reader needs to know whether what they just saw is good news.

Keeping runbooks alive

• Link the runbook in the alert. If the on-call has to search a wiki, the runbook does not exist.
• Update it in the postmortem. Every incident either confirmed the runbook or exposed a gap; the postmortem action item is the update.
• Game-day it twice a year. A junior engineer following the runbook alone in staging finds every stale command for you.
• Delete zombie runbooks. A wrong runbook is worse than none — it burns the reader's trust in all the others.

Runbooks are the interest payment on operational debt. Skip them and the debt compounds at incident time.