Prepare operations, and recognize the limits of a single host

Your application is running. If it stops answering tomorrow, do you know which alert you will receive, which command to run and which data you will be able to get back? This lesson has you prepare those answers on a Docker host. You measure the service from a client, you cause a health failure, you time a restore and you reboot the machine. You then record what comes back on its own and what needs an intervention. These tests do not prove the high availability of a single host.

What you will learn

You will link technical observations to operational decisions. Every result must be retrievable with a command.

• Write a service objective with an indicator, a target and a window.
• Link each alert to a Docker signal and to an intervention procedure.
• Tell a failing probe apart from a process that stops.
• Measure the age of a restorable backup and the duration of a restore.
• Check what comes back after the host reboots.
• Write a replayable runbook and recognize the limits of a single host.

Prerequisites

You must have completed these three lessons. Their procedures are reused here, without teaching restart policies or the backup of a database again.

• Limit resources and restarts of a Compose stack to understand unless-stopped, the deliberate stop and OOMKilled.
• Read logs and attribute saturation to relate logs, container state and consumed resources.
• Back up and restore an application to tell an archive that is merely present from a backup that is really restorable.

Work on a dedicated VM that you can reboot, never on a shared server. You need an account allowed to use Docker and sudo for the reboot, Docker Engine 29.9.2 and Compose 5.6.1 from the official repository, plus Bash, Python 3, curl, tar, sha256sum and the usual GNU tools. The measurements quoted were made on 4 November 2026 on Debian 13.7, on the reference VM, with these Docker and Compose versions.

The local ports 18764, 18765 and 18766 and the folders ~/atelier-vigie, ~/vigie-reprise and ~/sauvegardes-vigie must be free. The only download needed is the public Python image. The Python server serves the exercise, it is not meant to publish an application on the Internet.

Which application serves as the demonstration?

The vigie application serves a bulletin stored in a file. Its /healthz endpoint answers correctly as long as the indisponible file does not exist. Its /visites counter stays in memory. These are two different states: the bulletin must survive a reboot, the counter must not.

Create the dedicated folder. Leaving out -p makes mkdir fail if this folder already exists, rather than mixing the exercise with earlier files.

mkdir ~/atelier-vigie
cd ~/atelier-vigie
mkdir source donnees
printf 'station=ambre\n' > donnees/bulletin.txt

Save this program as source/serveur.py. It does not write to the data folder. Only you will create there the file that causes the failure.

import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Lock

visites = 0
verrou = Lock()


class Vigie(BaseHTTPRequestHandler):
    def repondre(self, code, corps, type_mime="text/plain; charset=utf-8"):
        octets = corps.encode()
        self.send_response(code)
        self.send_header("Content-Type", type_mime)
        self.send_header("Content-Length", str(len(octets)))
        self.end_headers()
        self.wfile.write(octets)

    def do_GET(self):
        global visites
        if self.path == "/healthz":
            if os.path.exists("/donnees/indisponible"):
                self.repondre(503, "indisponible\n")
            else:
                self.repondre(200, "ok\n")
        elif self.path == "/visites":
            with verrou:
                visites += 1
                valeur = visites
            self.repondre(200, json.dumps({"visites": valeur}) + "\n",
                          "application/json")
        elif self.path == "/bulletin":
            with open("/donnees/bulletin.txt", encoding="utf-8") as fichier:
                self.repondre(200, fichier.read())
        else:
            self.repondre(404, "inconnu\n")

    def log_message(self, format, *args):
        print(self.address_string(), format % args, flush=True)


ThreadingHTTPServer(("0.0.0.0", 8000), Vigie).serve_forever()

Then write compose.yaml. The port is published on the loopback interface of the VM, not on all its interfaces. The source and the data are mounted read-only. The PORT_VIGIE variable will let you replay the restore on another local port.

Two settings matter for the exercise. With restart: unless-stopped, Docker applies a restart policy: it starts the container again when its process stops, unless you stopped it yourself. The healthcheck block defines a health probe. Every five seconds, Docker runs a command inside the container that queries /healthz, and two failures in a row switch the state of the container to unhealthy.

services:
  vigie:
    image: python:3.13-slim@sha256:ae14d4ce8771606477a81446e7ef429edd6762305fc50c84507c1041fa8285d1
    command: ["python", "/app/serveur.py"]
    restart: unless-stopped
    read_only: true
    user: "65534:65534"
    environment:
      PYTHONDONTWRITEBYTECODE: "1"
    ports:
      - "127.0.0.1:${PORT_VIGIE:-18764}:8000"
    volumes:
      - ./source:/app:ro
      - ./donnees:/donnees:ro
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"
      interval: 5s
      timeout: 3s
      retries: 2
      start_period: 5s

In the terminal of the VM, define the reference that will also serve for the backup, then start the project. Keep this terminal open until the host reboots.

IMAGE=python:3.13-slim@sha256:ae14d4ce8771606477a81446e7ef429edd6762305fc50c84507c1041fa8285d1
docker pull "$IMAGE"
docker compose -p vigie config -q
docker compose -p vigie up -d --wait --wait-timeout 60 --pull never
curl --fail --max-time 2 http://127.0.0.1:18764/healthz
curl --fail --max-time 2 http://127.0.0.1:18764/bulletin

The two requests must answer ok then station=ambre. Here --wait waits for the health of the container; a mere presence in docker ps would not be enough.

If up --wait stops on container … is unhealthy and returns a non-zero code, read docker compose -p vigie logs. A probable cause is a umask of 077 when source/ and donnees/ were created: the container's account 65534 can no longer read the files. The logs then show can't open file '/app/serveur.py': [Errno 13] Permission denied and the container restarts in a loop (case replayed with cp -r under umask 077).

How do you measure the service from outside the container?

An SLO, for Service Level Objective, is a service objective. It combines a measured indicator, called an SLI (Service Level Indicator), a target and an observation window. The SLI of this exercise is the share of requests to /healthz that receive HTTP 200 before the client's maximum delay. Docker does not derive it from the running state. The guide SLOs, SLIs and error budgets details the method; a minimal example is enough here.

Here is the objective of the exercise: "During the window of this exercise, at least 99% of the twenty requests to /healthz must receive HTTP 200 in under two seconds." The requests start from the host, with a one-second pause between the end of one attempt and the start of the next. The 99% target is an exercise choice. With twenty requests, a single failure already gives 95%: this target tolerates none.

Save this script as mesurer-service.sh. It causes a failure during requests 8 to 13 so that you can see what the calculation counts. The failures stay in the output with their return code.

#!/usr/bin/env bash
set -eu
cd "$HOME/atelier-vigie"
ok=0
ko=0
date -u '+debut=%Y-%m-%dT%H:%M:%SZ'
for i in $(seq 1 20); do
  if [ "$i" -eq 8 ]; then touch donnees/indisponible; fi
  if [ "$i" -eq 14 ]; then rm donnees/indisponible; fi
  if code=$(curl --fail --silent --show-error --max-time 2 \
      --output /dev/null --write-out '%{http_code}' \
      http://127.0.0.1:18764/healthz); then
    rc=0
  else
    rc=$?
  fi
  if [ "$rc" -eq 0 ] && [ "$code" = 200 ]; then
    ok=$((ok + 1))
  else
    ko=$((ko + 1))
  fi
  printf 'requete=%s http=%s rc=%s\n' "$i" "$code" "$rc"
  sleep 1
done
date -u '+fin=%Y-%m-%dT%H:%M:%SZ'
printf 'succes=%s echecs=%s total=%s\n' "$ok" "$ko" "$((ok + ko))"
awk -v ok="$ok" 'BEGIN { printf "taux=%.1f%% cible=99%%\n", 100*ok/20 }'

Run it from the VM, therefore outside the container.

bash mesurer-service.sh

The measurement of 4 November 2026, from 14:25:16 UTC to 14:25:36 UTC, gave this result:

succes=14 echecs=6 total=20
taux=70.0% cible=99%

The six HTTP 503 responses made curl --fail return code 22. Restoring the service at the end does not erase these failures. The objective of this window is not met.

This client observes the published port, but it still runs on the same host. It can no longer measure once the VM goes down. In operations, the measurement must also come from a machine outside the host, over the real network path of the users. A local probe that answers proves neither the DNS, nor the TLS, nor a business operation that /healthz does not exercise. The definition of SLIs and SLOs in the Google SRE book likewise separates the observed indicator from the target set for it.

Why does a failing probe restart nothing?

A process can keep running while refusing to serve. The restart policy reacts to its stopping, not to a switch to unhealthy. The indisponible file will make the probe fail without killing Python. The lesson Startup, readiness and health already shows that an unhealthy service is not restarted; here, you prove it with RestartCount and you tie this state to the "Health" alert of the table below.

Write attendre-sante.sh to bound the waits. The script returns 1 if the expected state does not arrive; it does not turn an expired wait into a success.

#!/usr/bin/env bash
set -eu
conteneur=$1
attendu=$2
for tentative in $(seq 1 60); do
  etat=$(docker inspect -f '{{.State.Health.Status}}' "$conteneur")
  if [ "$etat" = "$attendu" ]; then
    printf 'sante=%s\n' "$etat"
    exit 0
  fi
  sleep 1
done
printf 'Etat attendu absent : %s ; dernier etat : %s\n' "$attendu" "$etat" >&2
exit 1

Read the counter and the PID, cause the failure, then read them again. ps -aq also finds a stopped container, unlike ps -q.

cd ~/atelier-vigie
C=$(docker compose -p vigie ps -aq vigie)
bash attendre-sante.sh "$C" healthy
docker inspect -f 'sante={{.State.Health.Status}} RestartCount={{.RestartCount}} Pid={{.State.Pid}} StartedAt={{.State.StartedAt}}' "$C"
touch donnees/indisponible
bash attendre-sante.sh "$C" unhealthy
sleep 20
docker inspect -f 'sante={{.State.Health.Status}} RestartCount={{.RestartCount}} Pid={{.State.Pid}} StartedAt={{.State.StartedAt}}' "$C"

During the measurement, RestartCount stayed at 0, including after the extra twenty seconds, and only the health had changed. The PID and StartedAt did not move either: a second replay read PID 3955 and StartedAt=2026-11-04T14:50:35.669721637Z, identical before and after the wait. Your values will be different; what matters is that they do not change. An identical PID shows that the process was not started again, and an unchanged StartedAt confirms it.

Look at the detail of the probe and the logs before deciding on an action. Here, you know the cause and you can remove the file.

docker inspect -f '{{range .State.Health.Log}}{{.ExitCode}} {{.Output}}{{end}}' "$C" | tail -n 15
docker logs --tail 10 "$C"
rm donnees/indisponible
bash attendre-sante.sh "$C" healthy
curl --fail --max-time 2 http://127.0.0.1:18764/healthz

The health log contains HTTP Error 503. Once the file is removed, the answer becomes ok again, without recreating the container. A blind restart would have kept this file, mounted from the host, and would therefore not have fixed the cause.

Which signals should trigger an intervention?

An alert must name what you observed and where to find the matching procedure. The signal does not always give the cause. Docker does not send alerts itself: it exposes signals that a monitoring tool reads, as in the guide Effective alerting (in French). This lesson decides what to watch; the choice of tool is yours. Start with a reading of the state and the consumption of Docker.

docker inspect -f 'sante={{.State.Health.Status}} RestartCount={{.RestartCount}} OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' "$C"
docker stats --no-stream "$C"
docker system df
RACINE_DOCKER=$(docker info -f '{{.DockerRootDir}}')
df -h "$RACINE_DOCKER"

In the measurement, the inspection gave healthy, RestartCount=0 and OOMKilled=false. docker system df counted one image of 178.4 MB and one container of 12.29 kB. The file system of /var/lib/docker had 17 G available out of 20 G, with 10% usage. These figures describe this host on that day; they are not recommended thresholds.

docker system df does not show the free space of the disk. It accounts for Docker objects. df looks at the file system, which can also hold your backups and other files. The reference of docker system df describes this scope.

The following table can become the entry table of your operations documentation. There is no universal threshold for the frequency of restarts or the remaining space: choose yours for your service, then test them.

Alert to define	Signal and command	Procedure entry and expected result
The service refuses requests	/healthz and .State.Health.Status in docker inspect	"Health": read the probe log and docker logs, fix the cause, get healthy and HTTP 200 back.
Restarts keep happening	Change in .RestartCount between two readings of the same container	"Repeated stops": keep the logs and the identity of the container, identify the stop before starting again. The counter must no longer increase during the verification window.
A process of the container was killed for lack of memory	.State.OOMKilled in docker inspect and the oom event of docker events --filter event=oom	"Memory": correlate the event with the logs and the configured resource limits, then measure with docker stats. Do not raise a limit without checking the capacity of the host.
The disk nears the decided limit	docker system df and df -h "$RACINE_DOCKER"	"Disk": identify what occupies the space, keep the data, remove only the objects known to be disposable and check the free space again.

Swipe to see more

RestartCount is not a counter of VM reboots. A recreation also produces a new container and a new counter. Keep the container identifier with your readings.

Likewise, OOMKilled=false describes the inspected state, not the absence of any memory incident in the past. Conversely, OOMKilled can be true on a container that is still running: the field switches to true as soon as a process of the container is killed for lack of memory, and it stays that way. An exit code 137 without OOMKilled or an oom event does not allow a conclusion: a forced stop by SIGKILL can produce it too. The lesson Cap memory and the number of tasks of a container measures these cases.

If the application or Docker no longer answers, the local collection can disappear as well. Have the host checked from outside, and pass the latest readings on to the person in charge of the recovery. An alert that leads only to "restart" is incomplete.

How do you time a complete restore?

The RPO, for Recovery Point Objective, sets the acceptable data loss, expressed here as a duration. The age of the latest restorable backup at the moment of the incident lets you compare your recovery with this objective. The RTO, for Recovery Time Objective, sets the acceptable recovery time. You must record a duration and its bounds before you can compare it with this objective.

The diagram places the two measurements on one timeline. Look at where each duration starts: the RPO is read before the incident, between the last restorable backup and the incident; the RTO is read after it, until the verified recovery.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

You choose these objectives with the people who use the service. No command guarantees them. The NIST definitions, in Recovery Point Objective and Recovery Time Objective, tie them to what the activity must be able to get back and to the delay it can bear.

What must be kept for this exercise?

You must keep the program, its configuration, the data and the exact image. The bulletin does not change during the copy and the program never writes it: this file copy is consistent in this particular case. It does not replace the logical backup of a database taught in Back up and restore an application.

An export by digest alone loses the name of the image. On Docker 29.9.2, docker image save accepts the pinned reference, but after docker load the output gives only Loaded image ID: sha256:…, docker image ls lists no image and docker image inspect on the pinned reference answers No such image. The block below therefore creates a local alias from the pinned reference, checks that it designates the same image, then exports both references. This alias is for transport; the containers stay configured with tag@sha256. Since docker image tag replaces an existing association carrying the same name, do not run this block on a host where python:3.13-slim serves another project.

(
  set -eu
  cd ~/atelier-vigie
  IMAGE=python:3.13-slim@sha256:ae14d4ce8771606477a81446e7ef429edd6762305fc50c84507c1041fa8285d1
  TAG=${IMAGE%@*}
  docker image tag "$IMAGE" "$TAG"
  test "$(docker image inspect -f '{{.Id}}' "$IMAGE")" = \
       "$(docker image inspect -f '{{.Id}}' "$TAG")"
  mkdir -m 700 ~/sauvegardes-vigie
  date -u +%s > ~/sauvegardes-vigie/instant-sauvegarde
  tar czf ~/sauvegardes-vigie/vigie-projet.tar.gz -C ~ \
    atelier-vigie/compose.yaml atelier-vigie/source atelier-vigie/donnees
  docker image save -o ~/sauvegardes-vigie/vigie-image.tar "$IMAGE" "$TAG"
  cd ~/sauvegardes-vigie
  sha256sum vigie-projet.tar.gz vigie-image.tar instant-sauvegarde > SHA256SUMS
  sha256sum -c SHA256SUMS
)

Each line of the check must end with OK. A checksum detects an alteration of these files, not an incomplete backup, nor a malicious actor who could also replace the checksum file. Only the restore will test the content.

How do you replay the recovery on a separate environment?

Create restaurer-vigie.sh in ~/atelier-vigie. It uses a new folder, a new project and port 18766. It fails if the target folder already exists or if the health and the bulletin are not found again. The duration is displayed only after these checks.

#!/usr/bin/env bash
set -eu
debut=$(date +%s%N)
cd "$HOME/sauvegardes-vigie"
sha256sum -c SHA256SUMS
mkdir "$HOME/vigie-reprise"
tar xzf vigie-projet.tar.gz -C "$HOME/vigie-reprise" --strip-components=1
docker load -i vigie-image.tar
IMAGE=python:3.13-slim@sha256:ae14d4ce8771606477a81446e7ef429edd6762305fc50c84507c1041fa8285d1
docker image inspect "$IMAGE" --format '{{.Id}}'
cd "$HOME/vigie-reprise"
PORT_VIGIE=18766 docker compose -p vigie-reprise up -d \
  --wait --wait-timeout 60 --pull never
curl --fail --max-time 2 http://127.0.0.1:18766/healthz
bulletin=$(curl --fail --silent --show-error --max-time 2 \
  http://127.0.0.1:18766/bulletin)
if [ "$bulletin" != 'station=ambre' ]; then
  printf 'Bulletin restaure inattendu : %s\n' "$bulletin" >&2
  exit 1
fi
printf 'donnees=station=ambre\n'
fin=$(date +%s%N)
awk -v debut="$debut" -v fin="$fin" \
  'BEGIN { printf "restauration=%.1f s\n", (fin-debut)/1000000000 }'

First replay the procedure without stopping the initial service. You test the backup without deleting what works.

bash ~/atelier-vigie/restaurer-vigie.sh

The expected result is a successful checksum check, a reloaded image, the service healthy, ok, then donnees=station=ambre and a duration. An error interrupts the script with a non-zero code. The separate project avoids taking the data of the initial service for restored data. It still shares the disk and Docker with it, though: this test does not prove a recovery after the loss of the VM. In operations, also keep the backup off the host and replay it on a separate machine.

The loading must also be tested with an image that is absent from the target. This check was made separately on the measurement VM: after removing the image reserved for the test, docker load made its pinned reference usable again with --pull never. The procedure above keeps the image cache shared by the same Docker. Do not remove an image used elsewhere to reproduce the absence check; use a blank target.

What is lost between the backup and the incident?

Add a write that comes after the backup. It will not appear in the restored data: this is the data loss that the RPO bounds. First remove the recovery project of the preventive test so that the second restore starts again from an absent folder.

(
  set -eu
  cd ~/vigie-reprise
  PORT_VIGIE=18766 docker compose -p vigie-reprise down
  cd ~
  rm -r -- vigie-reprise
)
printf 'mesure posterieure\n' > ~/atelier-vigie/donnees/journal.txt
date -u +%s > ~/sauvegardes-vigie/instant-incident
docker compose -p vigie -f ~/atelier-vigie/compose.yaml stop
python3 - <<'PY'
from pathlib import Path

dossier = Path.home() / "sauvegardes-vigie"
sauvegarde = int((dossier / "instant-sauvegarde").read_text())
incident = int((dossier / "instant-incident").read_text())
print(f"age_sauvegarde_a_incident={incident - sauvegarde} s")
PY
bash ~/atelier-vigie/restaurer-vigie.sh
test ! -e ~/vigie-reprise/donnees/journal.txt
printf 'ecriture posterieure absente de la reprise\n'
date -u +%s > ~/sauvegardes-vigie/instant-reprise
printf 'reprise_moins_incident=%s s\n' "$(( $(cat ~/sauvegardes-vigie/instant-reprise) - $(cat ~/sauvegardes-vigie/instant-incident) ))"

The last check proves here the absence of this write, not an arbitrary loss of all the data. The backed-up bulletin is still available on the recovery port. The last two lines read the instant of the verified recovery and subtract the instant of the incident, written just before the primary is stopped: this is the duration that the RTO must bound, to the nearest second since date -u +%s counts whole seconds.

Two replays, on 4 November 2026, gave the values below. The second was made after adding instant-reprise, which the first lacked. In both, the recovery folder was recreated at each test and the image was still in the cache of the same VM.

Measurement	Bounds and context	First replay	Second replay
Preventive restore	From the start of the script to the verified health and bulletin	6.5 s	not recorded
Backup age	From the instant recorded with the backup to that of the declared incident	26 s	18 s
Restore after incident	From the start of the second run of the script to the same checks	6.4 s	not recorded
Incident to verified recovery	From instant-incident, written just before the primary stops, to instant-reprise, written after the data validation	not recorded	17 s

Swipe to see more

The roughly ten seconds it takes to stop the primary are part of the last duration: timed separately, this stop lasted 10.3 s. Docker waits here for the end of the grace period before killing the Python program. This delay is not a human reaction time. The reading measures from the launch of the stop to the validation; it does not measure, request by request, the exact duration of the unavailability.

Take as exercise objectives an RPO of five minutes and an RTO of thirty seconds, from the launch of the stop to the validation of the data. The two backup ages (26 s then 18 s) and the 17 seconds between the incident and the verified recovery stay below these bounds. The age depends on how fast you chain the commands: record your own. The 6.4 seconds of the script alone would not be enough to check the second objective, because they leave out the stopping of the primary.

These objectives are teaching choices. A measurement on an unloaded VM, with a small backup that is already accessible, does not become a production commitment. It counts neither the call to the on-call person, nor the time needed to get a standby machine, nor the redirection of the users from port 18764 to port 18766. Choose your objectives with the people who use the service and do not rewrite them afterwards to declare success.

What comes back after the host reboots?

This test assumes that both restores are finished and that the recovery project still exists. The return of the daemon does not restore every service: you must examine the restart policies and the deliberate stops. Prepare a witness without a restart policy, then put vigie back into service.

cd ~/atelier-vigie
docker compose -p vigie up -d --wait --wait-timeout 60 --pull never
IMAGE=python:3.13-slim@sha256:ae14d4ce8771606477a81446e7ef429edd6762305fc50c84507c1041fa8285d1
docker run -d --name vigie-temoin --read-only --user 65534:65534 \
  -p 127.0.0.1:18765:8000 \
  -v "$PWD/source:/app:ro" -v "$PWD/donnees:/donnees:ro" \
  "$IMAGE" python /app/serveur.py
sleep 12
curl --fail --max-time 2 http://127.0.0.1:18765/healthz
for i in 1 2 3; do
  curl --fail --max-time 2 http://127.0.0.1:18764/visites
done
cd ~/vigie-reprise
PORT_VIGIE=18766 docker compose -p vigie-reprise stop
systemctl is-enabled docker
systemctl is-active docker
cat /proc/sys/kernel/random/boot_id

The witness must answer ok and the counter of vigie must reach 3 if you did not query it before. Docker was enabled and active on the measurement VM. The recovery project is stopped on purpose to compare its case with that of vigie. Keep the boot identifier of the host.

Reboot only the authorized dedicated VM. This command cuts the SSH connection; reconnect when the VM comes back.

sudo systemctl reboot

After reconnecting, do not start any container before the inventory. These commands let you see what came back on its own.

systemctl is-enabled docker
systemctl is-active docker
cat /proc/sys/kernel/random/boot_id
cd ~/atelier-vigie
docker ps -a --format '{{.Names}} | {{.Status}}'
C=$(docker compose -p vigie ps -aq vigie)
bash attendre-sante.sh "$C" healthy
docker inspect -f 'etat={{.State.Status}} sante={{.State.Health.Status}} RestartCount={{.RestartCount}}' "$C"
docker inspect -f 'etat={{.State.Status}} politique={{.HostConfig.RestartPolicy.Name}} OOMKilled={{.State.OOMKilled}}' vigie-temoin
curl --fail --max-time 2 http://127.0.0.1:18764/visites
curl --fail --max-time 2 http://127.0.0.1:18764/bulletin

The boot identifier must have changed. On the measurement VM, Docker came back active, vigie became healthy again and the bulletin was intact. The first reading of the counter answered {"visites": 1}. The in-memory state was gone. RestartCount was still at 0, though: it does not count host boots.

vigie was running before the stop and had unless-stopped: Docker started it again. The witness had no policy and stayed stopped. The recovery project had unless-stopped, but you had stopped it on purpose. The Docker restart policies describe this difference in particular. "A policy is present" therefore does not promise an automatic return.

Check the failure of the witness, then start it again manually.

curl --fail --max-time 2 http://127.0.0.1:18765/healthz
printf 'code=%s\n' "$?"
docker start vigie-temoin
sleep 2
curl --fail --max-time 2 http://127.0.0.1:18765/healthz

Before docker start, curl returned code 7, because the connection was refused. After the restart, the witness answered ok. The containers stopped during this test may also have shown code 137. The Python program is PID 1 of the container and has no handler for SIGTERM: the kernel does not apply the default action to it, it stays alive, and Docker kills it with SIGKILL after ten seconds. This code is not proof of an OOM. The lesson Test the contract of an image shows how to refuse an image that ignores SIGTERM.

What should the operations documentation contain?

A runbook, or operations documentation, tells the person who intervenes what to check, what to do and how to recognize the result. It also states the conditions under which they must stop and ask for help. The guide Runbooks (in French) details their structure. A runbook is not valid because it looks complete when read: it must be replayed, by another person or on a separate environment.

For vigie, write down the dependencies before the procedures: the Docker service, the pinned image, compose.yaml, the program in source/, the bulletin in donnees/, the read permissions of account number 65534, under which the container runs, and access to the backup. The only published access is local to the VM. There is no database or external service to wait for.

The alert table gives the entries "Health", "Repeated stops", "Memory" and "Disk". Add the recovery steps with their commands and their expected result. The commands below are those of the exercise; this is not a universal procedure for another service.

Step	Command or procedure to replay	Expected result and decision
Check the daemon after reboot	systemctl is-enabled docker then systemctl is-active docker	enabled and active. Otherwise, deal with the daemon start before restarting the application.
Check the health	C=$(docker compose -p vigie ps -aq vigie), then bash ~/atelier-vigie/attendre-sante.sh "$C" healthy, then curl --fail --max-time 2 http://127.0.0.1:18764/healthz	healthy, then ok. If the wait fails, look at the probe and the logs; do not announce the recovery.
Resume the witness that stayed stopped	docker start vigie-temoin then the request on its port	Port 18765 answers ok. The in-memory counter starts over.
Check the backup before recovery	sha256sum -c SHA256SUMS in the backup folder	All the checksums are OK. An error forbids going on with this copy.
Restore on the separate project	bash ~/atelier-vigie/restaurer-vigie.sh with an absent target folder	The script returns 0 only after the expected health and bulletin. Keep the duration and the bounds of the reading.
Check the business data	curl --fail --max-time 2 http://127.0.0.1:18766/bulletin	station=ambre. Health alone does not validate the data.

Swipe to see more

The recovery script was replayed in the separate project vigie-reprise, before then after the incident. The measurements table keeps the context of each duration. Also have someone report the steps where a person must guess a path, a permission or a value: each one points to a gap in the documentation.

This procedure does not simulate the loss of the physical disk. Nor does it check that another operator has the necessary access during an outage. These dependencies and the loss of the writes made after the backup must stay explicit.

When is a single host no longer enough?

All the containers of this exercise depend on the same machine. When it goes down, they stop together. A restart policy does not move a service to another host, and a backup copy on the same disk disappears with it.

The reboot and the restore show a recovery with an interruption. Recreating the only available container does not demonstrate an update without interruption. Adding containers on this same host does not remove this point of failure.

The need to place replicas on several hosts, to replace a lost host or to organize updates without interruption justifies studying an orchestrator, Kubernetes for example. You must also solve access to the service and the availability of the data. Adopting an orchestrator does not guarantee high availability on its own. The lesson on availability and durability (in French) explains why these two properties are not the same thing.

How do you clean everything up after the exercise?

The cleanup removes only the resources of this workshop. It runs no prune and keeps the image cache, which could serve another project. If you want to keep the measurements or the backup, copy them out of the folders below before removing them.

(
  set -eu
  cd ~/atelier-vigie
  docker compose -p vigie down
  docker rm -f vigie-temoin
  cd ~/vigie-reprise
  PORT_VIGIE=18766 docker compose -p vigie-reprise down
  cd ~
  rm -r -- atelier-vigie vigie-reprise sauvegardes-vigie
  docker ps -a --filter name=vigie --format '{{.Names}}'
  test ! -e ~/atelier-vigie
  test ! -e ~/vigie-reprise
  test ! -e ~/sauvegardes-vigie
)

The filtered list must no longer show any container of the workshop and the three folder checks must succeed. The python:3.13-slim alias created for the backup stays in the image cache, next to the pinned reference. docker image rm python:3.13-slim would then delete the whole image, pinned reference included: do it only if no other project uses it. The VM stays started; destroying it, if you wish, belongs to the life cycle of your lab environment, not to a command targeting all the Docker objects of the host.

Key points

A service that can be operated is described by its objective, its observations and its recovery procedures. The tests serve to confront these three elements, not to turn a good local measurement into a guarantee.

• The SLO combines an indicator, a target and a window. Count the failures from outside the container and also plan an observation from outside the host.
• An unhealthy state does not trigger the restart policy.
• Each alert must point to a precise signal and to a verifiable action.
• Compare the RPO with the age of the restorable backup and the RTO with a recovery duration whose bounds are explicit.
• The return of the daemon restores neither the containers without a policy nor the in-memory state. unless-stopped also respects a deliberate stop.
• A local restore and a successful reboot prove neither the failover nor the high availability of a single host.

Knowledge check

Check your decisions when facing a failing probe, Docker signals and recovery measurements. The quiz covers the notions taught here, not the memorization of the values of the VM.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →