Sunday, 2:47 a.m. The phone buzzes: "CRITICAL ALERT - Payment API". After 45 minutes of fumbling without a clear runbook, you realise it was a false positive. The following Wednesday, same alert, same time, but this time it is real: exhausted from Sunday's wake-up, you react more slowly.
This scenario is not inevitable. On-call (the period during which an engineer stays reachable to respond to a production incident) can be sustainable, even formative, if it is well designed. This page is for teams that already run services in production and want to structure their duty: rotation, alerts, runbooks, compensation and well-being, following the principles codified by Google's SRE (Site Reliability Engineering).
What you will learn
- Size an on-call rotation according to the size of your team
- Filter alerts to keep only the truly actionable ones
- Structure an effective runbook and a handoff without information loss
- Calculate fair compensation for the on-call constraint
- Train newcomers through progressive shadowing
- Track the metrics that tell you whether your on-call stays sustainable
Why on-call exists
A production system is alive: it receives traffic, processes data, talks to external services. And sometimes it runs into situations that automation alone cannot handle.
Automation excels at predictable problems. A service that exceeds its memory limit can be restarted automatically. A database approaching saturation can trigger auto-scaling. But some situations require human judgement: coordinating the response when several interdependent services are affected, assessing the risks of a rollback with complex side effects, or investigating a problem never seen before.
On-call exists to provide this human response capacity when automation reaches its limits. Without it, an incident at 2 a.m. waits until 9 a.m., and seven hours of degradation can be costly, in money as in reputation.
The principles of healthy on-call
Google SRE set principles so that on-call stays sustainable over the long term. These principles are not theoretical ideals: they are operational rules tested at the scale of billions of users and documented in the SRE Book, freely available.
The balance between load and rest
A person cannot be on-call "all the time". On-call is a constraint on personal life: staying available, sober, within reach of your laptop has a psychological cost that builds up over the shifts.
Google recommends that a person is on-call no more than 25 % of the time, with at least 50 % of the time spent on engineering work. In practice, this means about one week in four in a classic rotation. Below this threshold, the rotation stays sustainable; above it, fatigue builds up and errors multiply.
During a shift (the continuous period during which a person is on duty, usually 8 to 12 hours), the number of incidents should stay limited. The SRE Book sets a maximum of 2 incidents per 12-hour shift, and justifies the figure: handling an incident, from root cause analysis to the postmortem and the fixes, takes six hours on average. It may seem low, but each incident means stress, potential sleep interruption, and cognitive recovery time afterwards.
If your team regularly receives more than 2 incidents per shift, it is an alarm signal. The problem is not that the team "is not reactive enough": the problem is that the system is not reliable enough to be supported in these conditions. The investment must go into stability, not into human endurance.
After a particularly heavy shift, several incidents, a wake-up in the middle of the night, high stress, the on-call person should get at least one recovery day. No mandatory meetings the day after a sleepless night.
Actionable alerts only
Every alert that rings on an on-call person's phone should meet four strict criteria. An alert that does not meet them all has no place in a paging system: it generates noise, wears down the vigilance of the team, and ends up being ignored.
| Criterion | Description | Compliant example | Non-compliant example |
|---|---|---|---|
| Urgent | Requires action within minutes/hours | "Payment API down" | A problem that can wait until Monday |
| Actionable | The person knows what to do | "Runbook: wiki/payment-errors" | "High CPU on server-42" |
| Symptom-based | What the user sees | "Latency > 2s" | "CPU at 90 %" |
| Non-redundant | 1 incident = 1 alert | One alert per incident | 10 alerts for the same problem |
Up-to-date runbooks
A runbook is the step-by-step documentation to respond to a specific alert. Without a runbook, the on-call person improvises at 3 a.m., under stress, with a brain that is not at its best.
A good runbook has 4 sections:
-
Context: which service is affected, its role, the user impact. It helps to understand the severity and prioritise.
-
Quick diagnosis (first 5 minutes): which dashboard to check, which logs to look at, which metrics to verify. Goal: categorise the problem.
-
Corrective actions organised by diagnosis:
- If symptom X, do Y
- If symptom Z, do W
- Concrete, tested actions (e.g.
kubectl get pods -l app=payment)
-
Escalation criteria: when to call for backup, who to call, how to reach them.
The following diagram shows where the runbook comes in when handling an alert: the primary follows it, and if the incident is not resolved within the time the team has set, they escalate to the secondary and the experts, or even to an incident manager for a major incident.
Fair compensation
On-call is a real constraint on personal life. Being on-call over a weekend means giving up going hiking, having wine at a restaurant, being offline. This constraint deserves compensation.
Compensation models vary. Some organisations choose payment: a flat bonus per on-call week, or an hourly rate for the time actually spent on incidents. Others prefer time off in lieu: a few hours of recovery for each on-call night, for example. Google states in its SRE Book that it offers time off in lieu or straight cash compensation, capped at a proportion of overall salary. Most organisations combine both approaches.
But the best compensation, in the end, is a light on-call load. A team prefers a quiet on-call, few alerts, rare and well-documented incidents, to an on-call that is well paid but exhausting. Investing in the reliability of the system is the real way to respect on-call people.
Designing a rotation
The on-call rotation is how you spread the load among team members. A good rotation is predictable, fair and sustainable: it never relies on the one-off goodwill of a volunteer.
Team size sets the pace
The sustainable on-call pace depends directly on the number of people available to rotate in the team. A team that is too small goes back on-call too often, with no margin for absences or the unexpected.
| Team size | On-call frequency | Comfort | Notes |
|---|---|---|---|
| 4-5 people | 1 week in 4-5 | Tight | Little margin for holidays/sickness |
| 6-8 people | 1 week in 6-8 | Comfortable | Margin for the unexpected |
| 8+ people | Primary/Secondary system | Optimal | Built-in training |
The SRE Book quantifies this sizing for a week-long rotation with a primary and a secondary: holding the 25 % ceiling requires at least eight engineers on a single site, or six per site when two sites share the duty.
Primary and secondary
The two-level system has several advantages. The primary handles routine incidents, the documented ones that follow the runbooks and are resolved in a few minutes. The secondary steps in when the primary is overwhelmed (several simultaneous incidents), unreachable (network problem, personal emergency), or out of their depth (a complex incident beyond their skills).
The secondary role is also an excellent training tool. A newcomer can start as secondary, watch how the primary handles incidents, step in progressively, before becoming primary themselves.
Some organisations spread over several time zones adopt a follow-the-sun model: each site covers its local office hours, and nobody is ever woken up at night. With two sites about twelve hours apart, each covers its daytime hours and night duty disappears; the SRE Book indeed presents this split between sites as the way to avoid night shifts. It assumes teams on several continents, which reserves it for organisations of a certain size.
The structured handoff
The handover between two shifts is a critical moment. A sloppy handoff creates gaps in coverage and loses information.
A good handoff covers four points. First, ongoing incidents: are there unresolved problems, pending actions, threads to pick up? Then, the state of the systems: have there been recent deployments, configuration changes, observed degradations? Then special information: a planned maintenance window, a business event (sales, product launch), particular context to know. Finally, explicit confirmation: the outgoing on-call confirms the handoff is complete and the incoming on-call is operational.
This handoff can take 5 minutes or 30 minutes depending on what happened. It should never be skipped, even after a completely quiet shift: a written handoff, however brief, is better than none.
Reducing the on-call load
The goal is not to "survive" a hard on-call. The goal is to make on-call light. Each incident is an opportunity for improvement, not just a problem to close.
Eliminating noisy alerts
After each shift, review the alerts received. Those that required no action, just a check that everything was fine, are candidates for removal or conversion into a log. Those that resolved themselves within minutes may need a delay before firing, or a higher threshold. Those that come back several times a week without anyone fixing the cause deserve an investment in a permanent fix.
The goal is to reach a state where every alert that rings is a real alert requiring a real action. The on-call person must trust the alerting system: it is this trust, more than the volume of alerts itself, that determines whether on-call feels sustainable.
Automating repetitive responses
If the runbook says "restart the service", why does a human do it at 3 a.m.? Repetitive, risk-free actions are what SRE calls toil: manual, repetitive, automatable work that brings no lasting value. Our guide on eliminating toil details how to identify and reduce it methodically.
A service that crashes from lack of memory can be configured with an auto-restart. A message queue that builds up can trigger auto-scaling. A failing batch can be retried automatically before alerting a human.
Since 2025, a new layer of tools has been added to this automation: observability platforms include AI triage agents that enrich an alert, with correlated logs, traces and recent deployments, before it reaches a human. This reduces the time spent gathering context at the start of an incident. These tools are still young: they require explicit human supervision and do not spare you solid runbooks or well-filtered alerts upstream; they are complements, not replacements.
Automation does not remove on-call: it reserves it for cases where human judgement is truly needed.
Investing in reliability
Each on-call incident should generate an improvement item. A recurring bug justifies a fix ticket. Missing monitoring justifies an improvement in observability. A missing procedure justifies a new runbook. A recurring problem justifies investing in a permanent fix rather than a repeated workaround.
If your team spends its time putting out the same fires again and again, the problem is not the team, it is the lack of investment in prevention.
Training newcomers for on-call
On-call cannot be improvised. Throwing someone in at the deep end without preparation guarantees stress, errors, and potentially badly handled incidents.
Progressive shadowing
The proven method is shadowing: the new member watches then progressively takes the controls, at a pace that leaves time to absorb the context before taking on the responsibility.
In the first week, the newcomer watches. They receive the alerts at the same time as the primary, but do not intervene. They ask questions, take notes, understand the systems and procedures. The goal is to absorb the context without the pressure of responsibility.
In the second week, the newcomer takes the controls under supervision. They are the first to diagnose and propose actions, but an experienced member validates before execution. It is the time for mistakes without consequences: better to be wrong now, with a safety net.
In the third week, the newcomer is primary with an experienced secondary as backup. They handle routine incidents alone, but know they can escalate quickly if something is beyond them.
Simulation exercises
Google practises the Wheel of Misfortune, an exercise where past incidents are replayed in realistic conditions. A game master picks a past incident scenario, the team is alerted as if for real (pager, same simulated time), the on-call person diagnoses and responds while the game master simulates the system's responses. Then comes the debrief: what worked, what was missing, what needs improving?
These exercises prepare for real incidents without the risk of production impact. They also reveal gaps in the runbooks and in the team's skills, often before a real incident reveals them in the worst possible way.
On-call well-being
On-call, even when well designed, has an impact on well-being. Protective practices are needed, and they are the responsibility of management as much as of the individual.
Avoiding burnout
No back-to-back on-call: at least one week off between two shifts. The right to disconnect outside on-call must be real: no "just in case" messages, no informal requests. If the number of incidents suddenly explodes, it is an alarm signal for management, not an individual problem to absorb in silence.
Two levers weigh particularly on on-call fatigue: non-actionable alerts and the lack of recovery after a stressful incident. Addressing them first protects team retention more than a higher bonus does.
Respecting sleep
The SLO (Service Level Objective, the service level objective that sets how much degradation is tolerated before it becomes unacceptable) determines the real urgency. Our guide SLOs, SLIs and error budgets explains how to define it properly. If your SLO tolerates 30 minutes of degradation, the on-call person does not need to respond in 2 minutes. A realistic response time (15-30 minutes) means not sleeping with the phone under the pillow.
Geographically distributed teams can practise the follow-the-sun model described above to guarantee that nobody is woken up at night, provided they have teams in time zones far enough apart to take over from each other.
Post-incident support
After a stressful incident (major outage, business pressure, long duration), the on-call person deserves recovery time. No mandatory meetings the next day. A supportive debrief if the incident was traumatic. Explicit recognition of the effort made, which costs nothing and changes a lot in how the load is perceived.
Metrics for healthy on-call
How do you know if your on-call is sustainable? Tracking these key indicators over time lets you detect a drift before it turns into departures. The thresholds in the table are indicative benchmarks to calibrate on your context, not values published by Google.
| Metric | Healthy threshold | Alert threshold | Why it matters |
|---|---|---|---|
| Incidents / week | < 5 | > 10 | Permanent load = burnout |
| MTTR (time to resolve) | < 1h | > 2h | Incomplete runbooks or overly complex systems |
| % actionable alerts | > 80 % | < 50 % | Trust in the system |
| False positive rate | < 10 % | > 25 % | Needless wake-ups, loss of trust |
| Satisfaction score | > 7/10 | < 5/10 | Sustainability as perceived by the team |
Key points
- On-call is necessary for critical systems, but it must be sustainable for the people who carry it.
- 25 % maximum of an engineer's time on-call, with at least 50 % spent on engineering work.
- 2 incidents maximum per 12-hour shift: beyond that, the problem is the system, not the team.
- Every alert must be actionable, symptom-based, non-redundant and truly urgent.
- Runbooks that are up to date, tested, with a quick diagnosis and explicit escalation criteria.
- Fair compensation for the personal constraint, but the real priority remains a light load.
- Training (shadowing, Wheel of Misfortune simulations) prepares newcomers without throwing them into the unknown.
- Well-being (rest, disconnection, post-incident support) protects the teams already in place.
- AI triage helps enrich the context of an alert, but replaces neither runbooks nor human judgement.
A good on-call is a boring on-call: few alerts, quick resolutions, quiet nights. It is the sign of a reliable system and of a team that invests in the right direction.
Knowledge check
Check that you have the essentials of this guide. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Runbooks and incident response: from diagnosis to action in 5 minutes
A new full quiz with random questions
Work again only on the questions you missed
FAQ: frequently asked questions
The questions below cover the most frequent doubts about on-call: how long it can sustainably last, the rotation, compensation and alert handling.