Runbooks and incident response: from diagnosis to action in 5 minutes

A runbook is a step-by-step procedure that turns an alert into action: observed symptom → diagnosis → remediation → escalation if needed. Without a runbook, the on-call SRE at 3 a.m. opens the dashboard, looks at the alert, and spends 30 minutes figuring out what to do. With a runbook, they know in under 5 minutes: what the impact is, how to diagnose, what to fix, and who to call if that is not enough. This guide teaches you to write actionable runbooks, structure incident triage by severity, run blameless postmortems, and measure your progress with MTTD, MTTA and MTTR (time to detect, to acknowledge and to resolve).

It is the module's hands-on workshop: the Incidents and postmortems lesson sets the principles of incident management, the one on on-call organises the duty; here, you write the documents the on-call person will open at 3 a.m.

What you will learn

Runbook vs playbook: the difference between a procedure for one alert and a procedure for a type of incident
The standard runbook format: the 4 sections (symptom, diagnosis, remediation, escalation) plus the access prerequisites
Writing good practices: toolbox pod, error classification, copy-paste commands, standardised checks
60-second triage: the reflex protocol to assess impact, blast radius, last change, and choose the strategy
Mapping severity ↔ SEV: the relation between alert severity (critical/warning) and incident level (SEV-1/2/3)
Roles during an incident: incident commander, ops lead, scribe, communicator, who does what
Communication templates: copyable messages for Slack and the status page
Blameless postmortem: a complete structure with 5 categories of action items (prevention, detection, mitigation, remediation, resilience)
CI validation: automating the check of runbook_url and dashboard_url

Runbook vs playbook

Before diving in, let's clarify the terminology:

Type	Scope	Example	When to use it
Runbook	One specific alert	`PaymentErrorRateHigh.md`	Alert X just fired, what do I do?
Playbook	One type of incident	`database-outage-playbook.md`	The database is down, whichever alert reported it

A runbook is short, direct, tied to one specific alert. A playbook is broader: it covers an incident scenario (network saturation, DB outage, DDoS attack) and can reference several runbooks.

The problem: an alert without a runbook

Picture it: 3 a.m., your phone rings. PaymentErrorRateHigh : critical. You open the dashboard. The error rate is at 4 %. Now what? What is causing these errors? Is it the database, a recent deployment, an external provider? Should you roll back? Who should you notify?

Without a runbook, every incident is an improvisation. The on-call SRE must rebuild the diagnosis from scratch, under stress, with partial visibility. The result: an MTTR counted in hours instead of minutes, inconsistent actions from one incident to the next, and a team that burns out.

With a runbook, the SRE reads the procedure, follows the steps, and solves, or escalates, in a few minutes. Diagnosis is no longer an art reserved for experts: it is a documented procedure that any member of the on-call team can run.

The standard runbook format

An effective runbook is structured in 5 sections: the access prerequisites, then the 4 sections of the incident cycle.

Full structure

Each row answers a question the on-call SRE asks, in this exact order. Section 0. Prerequisites is the one most often forgotten, and yet it blocks everything else: a perfect runbook is useless if the on-call person lacks the access to run it.

Section	Question	Content
0. Prerequisites	What do I need to run this runbook?	kubectl access, Grafana, logs, traces, toolbox pod
1. Symptom	What is happening from the user's point of view?	Impact, affected metrics, error classification
2. Diagnosis	What is the probable cause?	Toolbox commands, ordered checklists, decision tree
3. Remediation	How do I fix it?	Actions per cause, rollback, verification after action
4. Escalation	What if I cannot solve it?	Who, when, through which channel, what information to pass on

Prerequisites: the often forgotten section

A "copy-paste" runbook is useless if the on-call SRE does not have the required access. List them explicitly, along with the tools and conventions:

## 0. Prerequisites

### Required access
- kubectl with the `production` context (RBAC: `sre-oncall` role)
- Grafana: https://grafana.example.com (SSO)
- Loki: via Grafana → Explore
- Traces: Tempo via Grafana → Explore

### Tools
- Toolbox pod: `kubectl -n tools run -it --rm debug --image=nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 -- bash`
- Feature flags: LaunchDarkly → "payments" project
- Rollback: ArgoCD → "payment-api" application

### Conventions
- Production namespace: `payments`
- Service: `payment-api`
- Main dashboard: https://grafana.example.com/d/payment-red
- Logs query: `{namespace="payments", app="payment-api"}`

Do not expect pg_isready, curl or dig in your application images, which are often minimalist. Use a toolbox pod with the diagnostic tools:

kubectl -n payments run -it --rm debug \
  --image=nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 -- bash

From there, you can dig postgres.payments.svc, curl http://payment-api:8080/health, nc -zv postgres 5432, etc.

Full example: PaymentErrorRateHigh (production-ready version)

This runbook is directly reusable: replace the service, namespace and tool names with yours. Note the numbering of the diagnosis subsections, from 2.1 to 2.5: the tables and remediation actions refer to them explicitly, which avoids rereading the whole document under pressure.

# Runbook: PaymentErrorRateHigh

**Alert**: PaymentErrorRateHigh
**Alert severity**: critical
**Owner**: payments team (@payments-oncall)
**Last updated**: 2026-02-05
**Dashboard**: https://grafana.example.com/d/payment-red

---

## 0. Prerequisites

### Required access
- kubectl `production` context (RBAC: `sre-oncall`)
- Grafana (SSO)
- LaunchDarkly (feature flags)
- ArgoCD (rollback)

### Toolbox
    kubectl -n payments run -it --rm debug --image=nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 -- bash

### Conventions
- Namespace: `payments`
- Deployment: `payment-api`
- Dashboard: https://grafana.example.com/d/payment-red

---

## 1. Symptom

Users see errors during payment. The error rate of the payment-api
service exceeds 1 % over the last 5 minutes.

**Impact**: failed payments, direct revenue loss.

### Error classification

Before diagnosing, identify the **error type**:

| Type | Symptom | Probable cause | Diagnosis section |
|------|----------|----------------|-------------------|
| **5xx** | Server errors (500, 502, 503) | Application bug, DB down, failing dependency | 2.1, 2.2, 2.3 |
| **4xx** | Client errors (400, 401, 403) | API change, broken auth | No infra action, escalate to dev |
| **Timeout** | Requests that never complete | Saturation, slow DB, network | 2.4 |
| **TLS/DNS** | Connection errors | Expired certificate, misconfigured DNS | 2.5 |

Check in the "Error breakdown" dashboard which type dominates.

---

## 2. Diagnosis

Follow in order, stop as soon as a cause is identified.

### 2.1 Recent deployment?

    kubectl -n payments rollout history deployment/payment-api

If deployed within the last 30 minutes → **rollback** (section 3.1).

### 2.2 Database reachable?

From the toolbox pod:

    # Start the toolbox
    kubectl -n payments run -it --rm debug --image=nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 -- bash

    # Test PostgreSQL connectivity
    nc -zv postgres.payments.svc 5432
    # If OK: "Connection to postgres.payments.svc 5432 port [tcp/postgresql] succeeded!"

    # Does DNS resolve?
    dig postgres.payments.svc

If the DB does not respond → escalate to the DBA team (section 4.2).

### 2.3 External payment provider?

**Isolate whether it is us or the provider**:

1. Check the status pages:
   - Stripe: https://status.stripe.com
   - Adyen: https://status.adyen.com

2. In the logs, look for the error pattern:

   ```plaintext
   {namespace="payments"} |= "stripe" or "adyen" | json | status >= 500
   ```

3. If the errors come **from the provider** (their 5xx, not ours):
   - Nothing to do on the infra side
   - Inform the product owner
   - Wait for the provider to resolve it

4. If the errors are **on our side** (timeouts, TLS, DNS towards the provider):

   From the toolbox:
   ```bash
   curl -v https://api.stripe.com/v1/health
   dig api.stripe.com
   ```

### 2.4 Resource saturation?

    kubectl -n payments top pods -l app=payment-api

**If OOMKilled**: do not scale blindly. Check first:
- Current requests/limits: `kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'`
- Sudden spike or slow growth? (memory dashboard)
- Application logs for a memory leak (heap, GC)

**If CPU > 90 %**: check **throttling** before scaling:
- cAdvisor dashboard: `container_cpu_cfs_throttled_seconds_total`
- If throttling is high with a degraded p99 → scale-up justified
- If CPU is high but no throttling → maybe just a normal spike

### 2.5 TLS / DNS / egress problem?

From the toolbox:

    # Test egress to an external dependency
    curl -v https://api.stripe.com/v1/health

    # Check the certificate
    openssl s_client -connect api.stripe.com:443 -servername api.stripe.com

    # Internal DNS
    dig kubernetes.default.svc

---

## 3. Remediation

### 3.1 Roll back the last deployment

    kubectl -n payments rollout undo deployment/payment-api

**Check**: the error rate drops below 1 % within 5 minutes.

### 3.2 Turn off a feature flag

If a recent feature flag is suspected:

1. LaunchDarkly → "payments" project
2. Turn off the suspected flag
3. Check the error rate

### 3.3 Restart the pods

    kubectl -n payments rollout restart deployment/payment-api

Useful for an inconsistent state (stale DB connections, corrupted cache).

### 3.4 Scale up (after checking saturation)

    kubectl -n payments scale deployment/payment-api --replicas=6

Only if section 2.4 confirmed real saturation.

---

## 4. Escalation

| Delay | Action | Contact |
|-------|--------|---------|
| T+0 | Acknowledge the alert, start diagnosing | On-call SRE |
| T+15 min | If unresolved → call the secondary | @payments-oncall-secondary |
| T+30 min | If unresolved → escalate to the lead | @payments-eng-lead |
| T+1 h | If unresolved → incident commander | @incident-commander |

### 4.2 DBA escalation

If the database does not respond:

- Contact: @dba-oncall
- Slack: #dba-incidents
- Information to pass on: result of `nc -zv`, errors in the pod logs

**Information to pass on in any escalation**:
- Incident start time
- Dominant error type (5xx, timeout, TLS...)
- Impact (% errors, estimated affected users)
- Diagnosis steps already done
- Hypotheses tested and results

Linking runbooks to alerts

Every Prometheus alert must carry standardised annotations. The effective alerting guide (in French) details how to write these symptom-based alerts; here, we focus on the links they carry:

annotations:
  runbook_url: "https://wiki.example.com/runbooks/payment-error-rate"
  dashboard_url: "https://grafana.example.com/d/payment-red"
  log_query_url: "https://grafana.example.com/explore?query={namespace=\"payments\"}"
  owner: "payments-team"

Annotation	Use
`runbook_url`	Direct link to the alert's runbook
`dashboard_url`	Main dashboard for visual diagnosis
`log_query_url`	Pre-filled Loki query for the logs
`trace_query_url`	Pre-filled Tempo query (optional)
`owner`	Owning team (@team or email)

When the alert fires, the SRE clicks on runbook_url and lands directly on the procedure.

CI validation of the links

A broken link is worse than no link. Add a validation to your CI pipeline, which fails the build rather than letting on-call discover the dead link in the middle of the night:

Gating logic:

Parse the Prometheus rule files (.yaml)
Extract each runbook_url and dashboard_url
For each URL: check that it returns HTTP 200
If a severity: critical alert has no runbook_url → build failure

Example script, tested with yq 4.52 (Mike Farah's version) on a compliant rule file and then on a faulty one, where it returns 1 and names the offending alert:

#!/usr/bin/env bash
# check-runbook-urls.sh : chaque alerte critical a un runbook, et chaque lien répond
set -uo pipefail

RULES_DIR="${1:-prometheus/rules}"
failed=0

for rule_file in "$RULES_DIR"/*.yaml; do
  # URLs de runbook et de dashboard déclarées dans les annotations
  while IFS= read -r url; do
    [ -z "$url" ] && continue
    status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$url")
    if [ "$status" != "200" ]; then
      echo "ÉCHEC : $url renvoie $status ($rule_file)"
      failed=1
    fi
  done < <(yq '.groups[].rules[].annotations | (.runbook_url, .dashboard_url) | select(. != null)' "$rule_file")

  # Alertes critical déclarées sans runbook_url
  while IFS= read -r alerte; do
    [ -z "$alerte" ] && continue
    echo "ÉCHEC : alerte critical sans runbook : $alerte ($rule_file)"
    failed=1
  done < <(yq '.groups[].rules[] | select(.labels.severity == "critical" and .annotations.runbook_url == null) | .alert' "$rule_file")
done

exit "$failed"

Triage by severity: the 60-second protocol

When an incident occurs, the first 60 seconds are critical. Here is the reflex protocol:

Mapping alert severity ↔ incident SEV

The severity of an alert (critical/warning/info) is not the same thing as the SEV of an incident (SEV-1/2/3). Severity describes the alert's configuration; the SEV describes the real impact observed.

Alert severity	Probable incident SEV	Example
`critical`	SEV-1 or SEV-2 depending on real impact	Critical alert but 2 % of users → SEV-2
`warning`	SEV-2 or SEV-3 depending on duration and impact	Warning lasting 2 h → can rise to SEV-2
`info`	No incident	Notification, no immediate action

Incident severity grid

This grid is meant to decide within seconds, not to describe a situation finely. The principle to remember: it is the number of affected users and the criticality of the service that set the level, never the technical difficulty of the problem. The Incidents and postmortems lesson shows a four-level grid; three are often enough for a team that is starting out, what matters being that the criteria are written down and shared before the crisis.

Level	Criteria	Examples	Who to mobilise	Response time
SEV-1	> 50 % of users affected, core service down, data loss, security breach	Checkout down, data leak, corrupted database	IC + full team + management	< 15 min
SEV-2	< 50 % of users, important feature degraded, workaround possible	Slow search, delayed notifications, 1 provider out of 2 failing	On-call team + engineering lead	< 30 min
SEV-3	Limited impact, secondary feature, no revenue impact	Marketing page failing, slow internal tool	On-call team only	< 4 h (business hours)

Impact × urgency assessment

When impact alone does not settle it, cross it with urgency, that is, the speed at which the situation degrades. A moderate problem that gets worse minute by minute mobilises more than a wider but stabilised one.

	High urgency (worsening fast)	Low urgency (stable)
High impact (core service, > 50 % users)	SEV-1	SEV-2
Moderate impact (secondary feature, < 50 % users)	SEV-2	SEV-3
Low impact (cosmetic, internal tool)	SEV-3	No incident, ticket

Roles during an incident

A SEV-1 incident with 5 people all doing the same thing is worse than an incident with 4 people in defined roles.

The 4 essential roles

The most important rule fits in one row of the table: the Incident Commander does not type commands. As soon as they put their hands on a terminal, nobody coordinates any more, and the response heads in several directions at once.

Role	Responsibility	Who
Incident Commander (IC)	Coordinates the response, decides on actions, manages priorities, declares the end of the incident. Does not type commands.	Senior SRE or engineering lead
Ops Lead	Carries out the technical actions: diagnosis, remediation, monitoring	On-call SRE or a dev from the owning team
Scribe	Notes in real time: actions, timestamps, decisions, links, hypotheses tested. Feeds the postmortem.	Dedicated team member
Communicator	Keeps management, customer support and the status page informed at regular intervals (15-30 min)	Engineering manager or dedicated member

For a SEV-3, the on-call SRE holds all 4 roles. For a SEV-1, separating the roles is essential. The SRE Book also describes a planning lead, in charge of the logistics of long incidents (shift changes, follow-up tickets): this role is added when the incident spills over a shift, whereas the scribe is useful from the very first minute.

Communication templates

Writing a message during an incident costs time and attention from people who have none left. The two templates below can be filled in in one minute. They do not say the same thing: the external version never mentions internal technical detail, or a presumed cause until it is confirmed.

Internal update (Slack incident channel)

Post an update every 15-30 minutes:

**Incident Update: [timestamp]**

**Impact**: Checkout failing for ~15 % of users
**Status**: Diagnosis in progress
**Mitigation**: Rollback of v2.4.1 in progress
**Next step**: Check the error rate after rollback
**ETA**: 10 min for confirmation
**Owner**: @alice-martin (IC) / @bob-dupont (Ops Lead)

External update (status page)

For users and customers:

**[Investigating] Payment difficulties**

We have identified a problem affecting payments on our
platform. Some users may encounter errors during
checkout.

**Affected services**: Card payment
**Workaround**: Retry in a few minutes or use PayPal
**Next update**: in 30 minutes

_Last updated: 14:45 UTC_

A typical incident flow

Here is the complete sequence, from the alert ringing to the postmortem. The MTTA and MTTR clocks stop at precise steps of this sequence, which explains why acknowledgement and the resolution declaration must be done in the on-call tool, not just announced out loud.

Detection

The alert fires. The MTTD clock covers the time between the real start of the problem and its detection.
Acknowledgement

The on-call SRE acknowledges the alert in the on-call tool. The MTTA clock stops here.
60-second triage

Apply the protocol: impact, blast radius, last change, strategy. Decide the SEV.
Mobilisation (if SEV-1/2)

Create a dedicated Slack channel (#incident-2026-02-07-payments). Assign the roles: IC, Ops Lead, Scribe, Communicator.
Diagnosis

Follow the runbook: check the probable causes in order. The scribe notes each action and result.
Remediation

Apply the fix. Mitigation first (rollback, disable flag), RCA afterwards. Check the return to normal.
Communication

Inform stakeholders that the incident is resolved. Update the status page. The MTTR clock stops here.
Postmortem

Shortly after stabilisation, typically within a few working days, write a blameless postmortem with typed action items. Update the runbook if needed.

Blameless postmortem

A postmortem is a written document that analyses an incident after its resolution. The goal: understand what happened and prevent it from happening again. The key word is blameless: focus on systems and processes, not on individuals.

Why "blameless"?

If an engineer made a mistake, it is because the system allowed them to make it. A deployment that breaks production is not the engineer's fault, it is the fault of the pipeline that did not detect the problem.

The "Postmortem Culture" chapter of the SRE Book makes it the condition for everything else: an engineer who does not fear reprisals feels safe to report the details, and it is these details that make it possible to fix the root causes.

The 5 categories of action items

Vague action items ("improve monitoring") will never get done. Put each action in a category:

Category	Goal	Examples
Prevention	Stop the error from happening	CI tests, DB schema validation, mandatory review
Detection	Detect faster if it happens again	New alert, SLO, dashboard, Watchdog
Mitigation	Reduce the impact when it happens	Automatic rollback, feature flags, safe autoscaling
Remediation	Fix the root cause	Bugfix, refactoring, design change
Resilience	Tolerate the failure	Timeouts, circuit breakers, bulkheads, retry with backoff

Postmortem template

This template covers the seven blocks expected of a usable postmortem. The timeline is the one that demands the most rigour, and paradoxically the least effort: if a scribe did their job during the incident, it is already written.

# Postmortem: [Descriptive incident title]

**Date**: 2026-02-07
**Duration**: 14:32 to 15:47 (1 h 15 min)
**Severity**: SEV-1
**Incident Commander**: Alice Martin
**Ops Lead**: Bob Dupont
**Scribe**: Charlie Durand
**Postmortem author**: Bob Dupont
**Status**: Action items in progress

---

## Summary

The payment-api service returned a 5xx error rate of 12 % for
1 h 15 between 14:32 and 15:47, affecting about 8,000 payment
transactions. Root cause: non-backward-compatible DB schema migration.

## Impact

- ~8,000 failed payment attempts
- Estimated revenue loss: ~€24,000
- 47 customer support tickets opened
- Status page in "major outage" for 1 h

## Timeline

| Time | Event |
|-------|-----------|
| 14:28 | Deployment of v2.4.1 with a schema migration |
| 14:32 | PaymentErrorRateHigh alert (critical) |
| 14:34 | Acknowledgement, runbook opened, SEV-1 triage |
| 14:35 | Channel #incident-2026-02-07-payments created |
| 14:38 | Diagnosis: recent deployment identified |
| 14:42 | kubectl rollback attempted |
| 14:45 | Rollback failed, irreversible DB migration |
| 14:50 | Escalation to the DB team |
| 15:05 | Fix identified: code compatible with old and new schema |
| 15:35 | Fix deployed (v2.4.2) |
| 15:47 | Error rate < 0.1 %. Incident resolved |

## Root causes

Schema migration v2.4.1: column `payment_method` renamed to
`payment_type`. Pods in the rolling update used the old name →
SQL errors.

**Contributing cause**: no DB schema backward-compatibility validation
in the CI pipeline.

## What worked well

- MTTD = 4 min (fast alert)
- The runbook pointed to "recent deployment"
- MTTA = 2 min

## What must be improved

- Application rollback insufficient if the DB migration is irreversible
- The CI pipeline did not detect the incompatibility
- Customer support communication: 25 min delay

## Action items

| Category | Action | Owner | Deadline |
|-----------|--------|-------|----------|
| **Prevention** | DB schema backward-compatibility check in CI | @platform | 2026-02-15 |
| **Prevention** | Document the expand-then-contract pattern | @dba | 2026-02-20 |
| **Detection** | Alert on column-related SQL errors | @payments | 2026-02-25 |
| **Mitigation** | Runbook: "DB rollback impossible" case | @payments-oncall | 2026-02-12 |
| **Resilience** | Feature flag to enable/disable the migration | @platform | 2026-03-01 |

## Lessons learned

Schema migrations must be **backward compatible** (expand then
contract): the old and new code must be able to coexist during
the rolling update.

The 5 postmortem mistakes to avoid

These five mistakes show up in most useless postmortems. The first two are enough to cancel the whole point of the exercise: without trust, nobody shares the details, and without a named owner, no action is carried through.

Mistake	Why it is a problem	Solution
Looking for a culprit	People hide details	Focus on systems and processes
Action items without an owner	"Improve monitoring" will never be done	Owner + deadline mandatory
Untyped action items	Vague tickets without direction	Categorise: prevention/detection/mitigation/remediation/resilience
No timeline	Impossible to understand the chronology	Minute by minute, from the scribe
Postmortem too late	Hazy memories after 2 weeks	Write within a few working days

Incident metrics: MTTD, MTTA, MTTR, MTBF

Incident metrics measure the effectiveness of your process.

The 4 essential metrics

These four indicators split the incident cycle into separately measurable segments. Each points to a different team and lever, which avoids concluding "we must do better" without knowing where to act.

Metric	Full name	What it measures
MTTD	Mean Time To Detect	Time between the start of the problem and its detection
MTTA	Mean Time To Acknowledge	Time between the alert and the acknowledgement
MTTR	Mean Time To Resolve	Time between detection and the return to normal
MTBF	Mean Time Between Failures	Average time between two incidents

Recommended MTTR convention

MTTR has several definitions (Resolve, Recover, Repair). Adopt a team convention and document it:

MTTR = return to an acceptable user experience (not "all systems perfect")

If users can pay again, the incident is resolved, even if the final fix is not deployed yet.

Targets and levers

The targets shown are common orders of magnitude, to adjust to the criticality of your business. The right-hand column is the most useful day to day: it tells you what to fix when a metric drifts, instead of merely noting the drift.

Metric	Target	If off target	Lever
MTTD	< 5 min	Alerts too slow	Improve thresholds, symptom-first alerts, Watchdog
MTTA	< 15 min	Team does not react	On-call schedule, paging tool, alert fatigue
MTTR	< 1 h (SEV-1)	Resolution too slow	Better runbooks, automation, training
MTBF	Increasing	Same recurring incidents	Track postmortem action items

MTTD depends as much on dashboards (in French) designed with the RED and USE methods as on alerts: a symptom visible on a well-designed dashboard is spotted before the alert even rings.

Anti-patterns

Here are the most frequent mistakes in teams setting up their first on-call system. The first six rows concern the writing of runbooks, the last four the running of the incident: the two families occur independently of each other.

Anti-pattern	Problem	Solution
"Catch-all" runbook	30 pages for 15 alerts	1 runbook = 1 alert
No prerequisites	SRE lacks access on the day	Explicit 0. Prerequisites section
Vague commands	"Check that the database works"	Exact commands, toolbox pod
Reflex scale-up	OOMKilled → scale without checking	First requests/limits, leak, GC
No error classification	Treating 5xx and 4xx the same way	5xx/4xx/timeout/TLS table
Runbook never tested	Broken procedure on the day	Quarterly game day
Postmortem = blame	Nobody reports the details	Blameless: systems, not individuals
Vague action items	"Improve monitoring"	Owner + deadline + category
No scribe	Timeline rebuilt from memory	Dedicated scribe in SEV-1
No communication	Management calls the SRE	Templates + regular updates

Workflow: setting up the process

These eight steps spread over several weeks and follow one another in order. Do not try to write every runbook at once: the first one serves as a template for the next ones, and step 4 makes forgetting impossible since the build fails as soon as a critical alert arrives without a runbook.

Inventory your critical alerts

List all critical alerts. They are your priority candidates.
Write the first runbook

Pick the most frequent alert. Write the 5 sections (prerequisites
- 4 incident sections). Have a junior proofread it.
Standardise the annotations

Add runbook_url, dashboard_url, owner to every alert.
Configure CI validation

A script that checks: HTTP 200 on every URL, runbook mandatory if severity: critical.
Define your severity grid

SEV-1/2/3 with clear criteria, severity ↔ SEV mapping, roles per level.
Formalise the roles

IC, Ops Lead, Scribe, Communicator. Communication templates.
Write your first postmortem

After the next incident, use the template with the 5 categories of action items.
Iterate every quarter

Runbook review, action item follow-up, game day to test the procedures.

Key points

Runbook = procedure for one alert (short, direct). Playbook = procedure for a type of incident (DB down, saturation) that groups several alerts
A runbook has 5 sections: prerequisites (access, toolbox, conventions) + symptom + diagnosis + remediation + escalation
Use a toolbox pod (nicolaka/netshoot) for diagnosis, do not expect the tools to be in your application images
Error classification: 5xx, 4xx, timeout, TLS, each type has a different remediation
60-second triage protocol: impact, blast radius, last change, strategy (mitigation first, RCA afterwards)
4 roles in SEV-1: IC (coordinates), Ops Lead (executes), Scribe (notes the timeline), Communicator (informs)
Blameless postmortem with 5 categories of action items: prevention, detection, mitigation, remediation, resilience
MTTR convention: return to an acceptable user experience (not all systems perfect). Document your definition.

Knowledge check

Check that you have the essentials of this guide. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

The timer starts when you click Start
Multiple choice, true/false and short answer questions
You can move between questions
Detailed results are shown at the end

Sources

The references below are the primary sources of the practices described on this page: Google's two SRE books for the postmortem culture, and the public guides from PagerDuty and Atlassian for incident response and its metrics.

Google SRE Book, Postmortem Culture: sre.google/sre-book/postmortem-culture
Google SRE Workbook, Postmortem Culture: sre.google/workbook/postmortem-culture
Google SRE Book, Example Postmortem: sre.google/sre-book/example-postmortem
PagerDuty, Incident Response Guide: response.pagerduty.com
PagerDuty, Postmortem Guide: postmortems.pagerduty.com
Atlassian, Incident Management Metrics: atlassian.com/incident-management/kpis/common-metrics

Runbooks and incident response: from diagnosis to action in 5 minutes

What you will learn

Runbook vs playbook

The problem: an alert without a runbook

The standard runbook format

Full structure

Prerequisites: the often forgotten section

Full example: PaymentErrorRateHigh (production-ready version)

Linking runbooks to alerts

CI validation of the links

Triage by severity: the 60-second protocol

Mapping alert severity ↔ incident SEV

Incident severity grid

Impact × urgency assessment

Roles during an incident

The 4 essential roles

Communication templates

Internal update (Slack incident channel)

External update (status page)

A typical incident flow

Blameless postmortem

Why "blameless"?

The 5 categories of action items

Postmortem template

The 5 postmortem mistakes to avoid

Incident metrics: MTTD, MTTA, MTTR, MTBF

The 4 essential metrics

Recommended MTTR convention

Targets and levers

Anti-patterns

Workflow: setting up the process

Key points

Knowledge check

Knowledge check

How it works

Score by domain

Summary

Strengths

Weak areas

Recommended next

Sources