Skip to content

Write a Step-by-Step Incident Runbook That Works Under 2 A.M. Pressure

Replace dense architecture essays with copy-paste diagnostic commands, clear P1/P2 severity matrices, safe mitigations, and exact escalation contacts.

Copy the prompt and paste it into the AI assistant you already use.

How to write runbooks for sleep-deprived humans, not calm afternoon readers

  1. 01

    Put 'Impact & First 3 Diagnostic Checks' in the first 10 lines

    At 2:15 a.m., nobody wants a history of how the service was architected in 2023; link the exact dashboard URL and copy-paste diagnostic query immediately.

  2. 02

    Prioritize 'Stop the Bleeding (Mitigate)' before deep Root Cause

    Document safe reversible mitigations first—rolling back the last deploy, rate-limiting one runaway tenant, or failing over—before chasing obscure edge-case logs.

  3. 03

    Highlight destructive 'DO NOT RUN' traps in a bold warning callout

    If restarting all worker pods simultaneously causes a database connection storm, put a prominent warning box right above the restart instructions.

  4. 04

    Define time-boxed Escalation Triggers ('Page Secondary if red at +20m')

    Junior responders often burn 90 minutes trying to be heroes; give explicit permission and obligation: 'If unmitigated after 20 minutes, page @Platform-Secondary immediately.'

How it works in practice

Tribal Slack guidance from Senior Engineer

When QueueDepthHigh fires, first check Grafana Queue panel. Usually 1 org flooding webhooks—run top-orgs query in Datadog, throttle that org_id via admin CLI to 50 rps. Do NOT purge the main DLQ or we drop billing events. If DB CPU >85% too, page Infra.

2 A.M.-Ready Incident Runbook Card

ALERT: `WebhookQueueDepthHigh` (Severity: P2 -> P1 if age >15m) • USER IMPACT: Outbound customer webhooks delayed; core app & billing unaffected. • STEP 1 — DIAGNOSE (2 min): - Check [Grafana Queue Dashboard] + [DB CPU Panel]. - If DB CPU >85% -> STOP & Page `@infra-oncall` immediately. - Else run Datadog Saved View `Top-10-Webhook-Senders-5m`. • STEP 2 — MITIGATE SINGLE NOISY TENANT: - Run: `ops-cli rate-limit --org=<ID> --rps=50 --ttl=2h` -> Queue should slope down within 3 min. • ⚠️ NEVER DO THIS: Never purge `prod-webhook-dlq`—contains unbilled usage events.

Separating the DB CPU branch upfront and flagging the destructive DLQ trap saves critical minutes during high-stress triage.

Common questions

Where should operational runbooks live if our internal wiki goes down during an outage?

Mirror critical P1 runbooks and emergency escalation contact lists in an out-of-band repo or offline-synced folder independent of your primary SSO or hosted wiki.

How do we make sure incident runbooks stay updated after architecture changes?

Make 'Update or create runbook link attached to alert payload' a mandatory closing checklist item on every post-incident review and service launch.

Should non-engineering teams like Finance, Payroll, or PR also have incident runbooks?

Yes—documented crisis runbooks for wire-fraud phishing attempts, delayed payroll runs, or leaked press embargoes prevent costly improvisation under stress.

Learn more

Related guides and background articles for this task