It is 3 a.m. Your pager goes off. The site is down. You log in urgently, several colleagues do the same. Everyone investigates on their own, applies fixes without coordination. Nobody knows what the others are doing. After two hours of chaos, someone finds the problem by chance.
This page explains how to structure incident response with clear roles (incident commander, ops lead) and how to run blameless postmortems that turn every outage into learning. It is for DevOps and SRE teams that run services in production, whatever their current maturity level.
What you will learn
- Classify an incident by severity and know when to declare it
- Structure the response with clear roles (incident commander, ops lead, communications)
- Distinguish mitigation from resolution to restore the service as fast as possible
- Run a blameless postmortem with a proven structure and timing
- Turn corrective actions into tracked deliverables, not wishful thinking
- Avoid the traps that kill the post-incident learning culture
What is an incident?
Definition and distinction
An incident is an event that degrades or interrupts a service in a way users can see. It differs from a simple bug or anomaly by its immediate impact and its need for rapid intervention.
The boundary is not always obvious. A simple rule: if the event requires urgent human action to restore the service, it is an incident. A bug waiting patiently in a backlog is not one.
Classification by severity
Most organisations classify incidents by severity level. This classification determines who is mobilised and what urgency is given, and it must be defined before the crisis, never during it.
| Level | Common name | Definition | Example | Expected response |
|---|---|---|---|---|
| SEV1 | Critical | Service fully unavailable or data corrupted | Site unreachable, data loss | Immediate mobilisation, external communication |
| SEV2 | Major | Critical feature degraded for all users | Payments failing, slow authentication | Rapid mobilisation, communication to stakeholders |
| SEV3 | Moderate | Secondary feature affected, or critical feature for a subset | Broken PDF export, regional slowness | Handled within the following hours |
| SEV4 | Minor | Limited impact, workaround available | Display bug, degraded feature with an alternative | Normal handling, no urgency |
When to declare an incident?
The "Managing Incidents" chapter of the SRE Book recommends declaring an incident early rather than late. It is easier to de-escalate a minor incident than to catch up on a major incident not declared in time.
The SRE Book offers three trigger questions (is a second team needed? is the outage visible to users? does the problem resist an hour of analysis?), to which you can add the risk of escalation:
- The problem requires the intervention of several teams
- The problem is visible to users
- The problem is not solved after an hour of investigation
- The problem could get worse if left untreated
The lifecycle of an incident
A well-managed incident follows a predictable cycle. Each phase has its goals and owners, which avoids the grey areas where nobody knows who should act.
-
Detection
The incident is discovered, either by an automatic alert (monitoring), by a user report, or by direct observation. The faster the detection, the smaller the impact.
-
Triage
A quick assessment of severity and impact. Who must be mobilised? How urgent is it? This phase should not last more than a few minutes.
-
Mobilisation
The right people are contacted and an incident commander is appointed. A communication channel is set up (chat, call, war room).
-
Mitigation
The goal is to restore the service, not to understand the problem. Rollback, restart, failover, traffic redirection: anything that limits user impact.
-
Resolution
Once the service is restored, you can take the time to understand and fix the root cause. This phase can take hours or days.
-
Closure and postmortem
The incident is officially closed. A postmortem is scheduled to analyse what happened and identify improvements.
Structured incident management
The problem with unstructured incidents
Here is what typically happens during an incident without structure:
- Several people investigate in parallel without coordination
- Changes are made to the system without everyone knowing
- Managers bombard the team with questions, interrupting the work
- Nobody knows who decides what
- External communications are inconsistent or missing
- Stress rises, errors pile up
The SRE Book devotes a whole story to this kind of "unmanaged" incident: everyone acts alone, the fixes contradict each other, and the outage lasts much longer than it should. The DORA (DevOps Research and Assessment) reports show how wide the gaps between teams are: in 2024, the best performers recover from a failed deployment in less than an hour, the lowest performers in one week to one month. A clear response structure is one of the cheapest levers to narrow this gap.
Roles in incident management
The SRE approach defines clear roles, inspired by the emergency incident management system (ICS, Incident Command System) used by firefighters and emergency services since the 1970s.
The diagram shows how these roles fit together: the incident commander splits the work between three leads, and only the ops lead touches the system, which avoids concurrent changes.
Responsibility: overall coordination of the incident response.
What they do:
- Keeps an overview of the situation
- Assigns roles and delegates tasks
- Makes strategic decisions (escalation, external communication)
- Does NOT do the technical work themselves
Profile: someone who stays calm under pressure, able to synthesise and decide quickly. Not necessarily the strongest technical expert.
Typical line: "OK, Mary you keep investigating the logs, Robin you prepare a rollback just in case. I'll update the stakeholders in 15 minutes."
Responsibility: carrying out technical actions on the system.
What they do:
- Investigates the problem
- Applies fixes and mitigations
- Reports findings to the incident commander
- Is the only person allowed to change the system during the incident
Profile: technical expert of the system concerned. Able to debug under pressure.
Typical line: "I found that service X has stopped responding since the 2 p.m. deployment. I suggest a rollback."
Responsibility: handling all communications.
What they do:
- Writes updates for internal stakeholders
- Coordinates with external communication teams if needed
- Keeps the incident document up to date
- Filters information requests to protect the technical team
Profile: able to translate technical matters into business language. A good writer, organised.
Typical line: "I just sent an update to the VP. Next communication in 30 minutes unless something major changes."
Responsibility: logistics support and planning.
What they do:
- Handles practical issues (meals if the incident lasts, shift changes)
- Prepares handoffs if the incident spans several shifts
- Creates follow-up tickets
- Notes deviations from the normal state for the rollback
Profile: organised, attentive to detail, able to think in the medium term while the others handle the emergency.
Typical line: "The incident has been going on for 4 hours, I'm preparing a handover with the night team."
The living incident document
During an incident, a shared document (Google Docs, Notion, wiki, or a dedicated tool such as incident.io, PagerDuty, Rootly) must capture the state in real time. It is part of a team's operational documentation (in French), just like runbooks:
- At the top: current severity, incident commander, status (ongoing/resolved)
- Summary: 2-3 sentences on what is happening
- Impact: affected users, degraded metrics
- Timeline: chronology of events and actions (with timestamps)
- Ongoing actions: who does what
- Decisions made: for reference and the postmortem
This document serves as the "source of truth" during the incident and as the basis for the postmortem afterwards. The more the timeline is captured in real time, the less the postmortem requires laborious reconstruction after the fact.
The handoff: passing the baton cleanly
If the incident lasts longer than a shift (usually 8-12 hours), you must organise a handoff: a transfer of responsibility to a new team.
The handoff must be explicit:
- A full briefing on the current situation
- A formal transfer of the incident commander role
- Verbal confirmation: "You are now the incident commander, OK?" then "Understood, I have the lead."
- Communication of the change to the whole team
A sloppy handoff can undo hours of work if the new lead lacks the context.
AI in incident response: a tool, not a replacement
More and more incident management tools include AI-assisted features: automatic summary of the timeline, suggestion of probable causes based on past incidents, or a first draft of the postmortem. These capabilities reduce the writing load and speed up the initial triage.
This gain has a measured limit: the 2026 Catchpoint SRE Report finds a median operational toil (repetitive manual work with no lasting value, see the page on eliminating toil) of about 34 %, unchanged despite the massive adoption of AI. AI speeds up information gathering, it does not replace a clear role structure or a culture that pushes teams to actually fix the systemic causes identified in postmortems.
The postmortem: learning without blame
Why postmortems are essential
An incident resolved is not an incident finished. Without analysis, the same causes will produce the same effects. The postmortem (also called a post-incident review or, at some companies such as AWS, a correction of error) is the mechanism by which an organisation learns from its failures.
The goals of a postmortem:
- Document what happened (organisational memory)
- Understand the root causes (not just the symptoms)
- Identify concrete improvements (prevention)
- Reinforce what worked well (good practices)
Blameless culture: an absolute requirement
The concept of the blameless postmortem is fundamental. The idea: humans make mistakes, it is inevitable. What matters is designing systems that tolerate these errors or make them impossible.
What "blameless" means:
- You analyse actions, not people
- You assume everyone did their best with the information available
- You look for systemic flaws, not culprits
- You ask "why did the system allow this error?" rather than "who made this error?"
What "blameless" does not mean:
- Ignoring repeated problematic behaviour
- Not identifying who did what (the timeline must stay factual)
- Avoiding difficult discussions
When to trigger a postmortem?
Define your criteria before an incident occurs. Common criteria:
- A SEV1 or SEV2 incident
- Data loss, even minor
- Manual intervention needed (rollback, failover)
- Resolution time above a threshold (for example 1 hour)
- Monitoring failure (incident discovered manually)
- A near miss: an incident avoided by luck rather than by design (for example a disk at 99% noticed by chance during another task, minutes before it filled up)
- A stakeholder's request
Structure of a postmortem
A good postmortem follows a predictable structure. Rather than looking for a single root cause, you list the contributing causes (systemic factors: processes, tooling, documentation), because a production failure almost always has several factors. The guide on writing effective postmortems (in French) provides ready-to-use templates.
| Section | Content | Example |
|---|---|---|
| Summary | 2-3 sentences describing the incident | "On 15 January, the payment service was unavailable for 45 minutes after a faulty deployment." |
| Impact | Quantified impact metrics | "1,200 users affected, 340 failed transactions, 32 % of the monthly error budget consumed." |
| Timeline | Factual chronology with timestamps | "14:23, v2.3.1 deployed to production. 14:31, first latency alert. 14:35, incident declared..." |
| Contributing causes | Analysis of systemic factors | "The deployment contained a regression in DB connection handling. Integration tests did not cover this scenario." |
| What worked | Positive elements to reinforce | "Monitoring detected the problem in 8 minutes. The rollback was done in under 5 minutes." |
| What can be improved | Weaknesses identified | "No load test before deployment. No canary deployment." |
| Actions | Concrete items with owner and deadline | "Add a DB regression test (Alice, 22 January). Implement canary deployment (Bob, 15 February)." |
The postmortem meeting
The postmortem is not just a document, it is also a meeting where participants discuss the incident out loud, which often brings out details missing from the written timeline.
Timing: shortly after the incident is stabilised, typically within a few working days, while the facts and context are still fresh. There is no universal deadline: Atlassian holds the meeting within 24 to 48 hours and at most five working days, PagerDuty within three calendar days for a SEV-1 and five working days for a SEV-2. These are organisations' practices, and the Google SRE Book sets no numeric deadline.
Participants: everyone who took part in handling the incident, plus people who could shed light on it (product team, technical experts).
Agenda:
- Reading the timeline (recalling the facts)
- Discussing the causes (no blame)
- Brainstorming improvements
- Prioritising and assigning actions
- Validating the final document
Rules:
- No interruptions while the timeline is read
- Clarifying questions welcome
- No "you should have...", rephrase as "the system could have..."
Actions: the real deliverable
A postmortem without actions is a futile exercise. Actions are the real deliverable of the process, not the document itself.
Each action must have:
- A clear description of what must be done
- An owner responsible for completing it
- A realistic deadline
- Follow-up until completion
Sharing the learning
A postmortem that stays in a drawer is useless. Google actively encourages sharing, in particular through an exercise called the Wheel of Misfortune: new SREs replay a past postmortem, each playing one of the roles described in the document, to practise incident response.
Several channels spread postmortems beyond the team concerned:
- Central repository: all postmortems accessible to the organisation
- Monthly newsletter: a "postmortem of the month" highlighting an interesting case
- Reading clubs: sessions where one team analyses another team's postmortem
- Onboarding: newcomers read recent postmortems to understand the systems
Cultivating the postmortem culture
Common obstacles
Introducing a blameless postmortem culture is not easy, even with the best will in the world. Here are the frequent obstacles encountered and how mature organisations get around them.
| Obstacle | Symptom | Solution |
|---|---|---|
| Fear of blame | People downplay their mistakes | Leadership by example, reward transparency |
| Lack of time | "We don't have time for meetings" | Build it into the standard process, limit the duration |
| Untracked actions | The same problems come back | Track actions, regular review |
| Shallow postmortems | "It was a bug, we fixed it" | Train in analysis techniques (5 Whys, etc.) |
| No perceived value | "It's pointless" | Show the incidents avoided thanks to past actions |
The role of leadership
The blameless culture must come from the top. If a manager punishes someone after a postmortem, the whole team's transparency effort is ruined in one go.
Leadership must:
- Take part in postmortems (show that it matters)
- Reward transparency, not perfection
- Protect teams from demands to "find a culprit"
- Celebrate learning, not hide failures
At Google, a 2014 TGIF, the company-wide weekly meeting hosted by its founders, was devoted to postmortems: a clear message about the importance of this practice.
Rewarding good practices
Counter-intuitively, you must reward those who declare incidents and write quality postmortems.
- Peer recognition for well-written postmortems
- Recognition in performance reviews
- Public sharing of good examples
- Never a sanction for causing an incident (except repeated negligence)
Common pitfalls
These pitfalls come back in many experience reports published by SRE teams, whether small startups or large platforms. Knowing them in advance lets you avoid them rather than rediscover them the hard way.
| Pitfall | Why it is a problem | How to avoid it |
|---|---|---|
| Looking for a culprit | Kills transparency, problems get hidden | Train in blameless culture, leadership by example |
| Postmortem too late | Memory fades, details are lost | Systematically schedule the meeting within a few working days |
| Actions without an owner | Nothing gets done, same incidents | Each action has an owner and a deadline |
| No action follow-up | Forgotten actions, recurring problems | Weekly review of open actions |
| Document never shared | Learning lost | Central repository, newsletter |
| Incomplete timeline | Analysis impossible | Document in real time during the incident |
Key points
-
Structure incident management
Clear roles (incident commander, ops lead, communications) reduce chaos and speed up resolution.
-
Mitigation first, resolution second
Restore the service quickly, even with a temporary solution. Understanding the problem can wait.
-
Blameless postmortems, always
You analyse systems, not people. Humans make mistakes, systems must tolerate them.
-
Look for contributing causes, not a single root cause
An incident almost always results from several systemic factors that lined up.
-
Actions are the real deliverable
A postmortem without concrete actions (owner + deadline) is a futile exercise.
-
Share the learning
A postmortem in a drawer is useless. Make them accessible, discuss them, learn collectively.
-
AI helps triage, it does not replace culture
AI-assisted tools speed up information gathering, but only a clear role structure and rigorous action follow-up durably reduce operational toil.
Knowledge check
Check that you have the essentials of this guide. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: On-call: organising production duty
A new full quiz with random questions
Work again only on the questions you missed
External resources
Google's SRE Book remains the most cited reference on these practices: the chapters listed below detail incident management and the postmortem culture with a level of precision beyond the scope of this page.
- Managing Incidents (Google SRE Book), structured incident management
- Postmortem Culture (Google SRE Book), building a learning culture
- Example Postmortem (Google SRE Book), Google's postmortem template
FAQ: frequently asked questions
The questions below cover the most frequent doubts about incidents and postmortems: when to trigger them, their structure, the role of the incident commander and the blameless culture.