Limit resources and restarts of a Compose stack

A Compose service with no limit can swallow all the memory of the machine, and a service that crashes stays stopped until someone starts it again. This lesson is for a beginner who can already write a stack: you set memory, CPU and process limits, then you choose what Docker does when a container stops.

Above all, you will see, on containers that stop in four different ways, which ones the restart policy restarts and which ones it leaves stopped. You finish with the grace period, the time Docker gives a service to stop cleanly before killing it.

What you will learn

• Limit the memory, CPU and number of processes of a Compose service, and check that Docker applies these limits.
• Recognize a container killed for lack of memory with OOMKilled.
• Choose a restart policy with restart.
• Tell what the policy restarts (a process that dies by itself) from what it does not restart at that moment (docker stop, docker kill).
• Restart or relaunch a service by hand.
• Shorten or lengthen the grace period with stop_grace_period.

Prerequisites

You need a Linux host with Docker Engine 29.9.2 and its Compose 5.6.1 plugin, installed from the official repository as shown in the lesson on installing Docker. The images are pinned by their digest. The kernel files read below (/sys/fs/cgroup/...) are those of cgroup v2, the version that Debian 13 uses.

You know how to write a stack thanks to the lesson Describe an application with Compose, and how to read the health of a service thanks to the previous lesson, Startup, readiness and health. This lesson uses docker compose ps and docker inspect without explaining them again.

Two neighboring lessons detail what this page only summarizes. The four restart policies of docker run are in Read a Docker startup failure, and the stop signals in Container lifecycle, PID 1 and clean shutdown. If you want to understand how the kernel enforces limits, Cap a container explains it, and Namespaces and cgroups: measure a container's isolation details the cgroups; you do not need them here.

You work in a ~/budgets folder, which the lesson has you create. Its name becomes the project name: the containers are called budgets-....

How do you limit the memory, CPU and processes of a service?

With three keys at the service level: mem_limit for memory, cpus for CPU and pids_limit for the number of processes. A limit is a hard ceiling: the container cannot exceed the memory or the CPU it sets, whatever the program does. For memory, swap (the memory exchanged with the disk) comes on top of this ceiling as long as memswap_limit does not bound it, as the section on exceeding the memory shows.

Create the folder and a first compose.yaml with a single service, indexeur, which only waits.

mkdir -p ~/budgets
cd ~/budgets

services:
  indexeur:
    image: alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
    command: ["sleep", "infinity"]
    mem_limit: 48m
    cpus: 0.25
    pids_limit: 20

mem_limit: 48m sets the maximum memory to 48 MiB, cpus: 0.25 allows a quarter of a core and pids_limit: 20 allows at most twenty processes. These values are small so that the demonstration reaches them quickly. In a real stack, you choose them from the consumption measured on the service.

Before starting, ask Compose for the resolved version of the file. This is what Docker will really receive.

docker compose config | grep -E 'mem_limit|cpus|pids_limit'

    cpus: 0.25
    mem_limit: "50331648"
    pids_limit: 20

Check that the three limits are present, and that the memory is converted to bytes (48 MiB is 50,331,648 bytes).

This command also acts as a safeguard. A limit written at the wrong indentation level is not silently ignored: with Compose 5.6.1, docker compose config stops with exit code 1 and a message that names the faulty key. For example, mem_limit slipped under deploy gives the following message, preceded by the word validating and the path of the file.

services.indexeur.deploy additional properties 'mem_limit' not allowed

Read the path before the word additional: it points to the place where the key was wrongly put, here services.indexeur.deploy. A key put at the services level gives services.mem_limit must be a mapping, and a key that overflows its line gives a YAML syntax error (did not find expected key). Fix the indentation before going further.

Is there another way to write these limits?

Yes: the deploy.resources.limits section. It comes from the Compose specification, which other orchestrators also read, and you will meet it in files written for them. With Compose 5.6.1 on Docker installed on your machine, mem_limit at the service level and deploy.resources.limits.memory are both applied. The following two spellings therefore give the same memory limit.

    # Short form, at the service level
    mem_limit: 48m

    # Specification form, under deploy
    deploy:
      resources:
        limits:
          memory: 48M

The same section also accepts cpus: "0.25" and pids: 20 under limits. A service written this way, started with Compose 5.6.1, gave the same values as the short form in docker inspect (memory=50331648 cpu=250000000 processes=20) and in the kernel files (50331648, 25000 100000 and 20), which you learn to read just below. A limit put in the wrong place under deploy is refused with a message, as with the short form.

Choose one or the other, not both, and use the same one throughout the file. This lesson uses the short form, which is easier to read. A deploy.resources.reservations key reserves a minimum, and a relative weight such as cpu_shares shares the CPU when the machine is busy: neither of them caps the container.

How do you check that Docker applies the limits?

By reading two places: the configuration that Docker recorded, then what the kernel really enforces. The Linux kernel enforces limits with its cgroups (control groups) and exposes them in the container as files, in /sys/fs/cgroup/. Start the service and compare.

docker compose up -d indexeur
docker inspect -f 'memory={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}} processes={{.HostConfig.PidsLimit}}' budgets-indexeur-1
docker compose exec indexeur sh -c 'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max /sys/fs/cgroup/pids.max'

memory=50331648 cpu=250000000 processes=20
50331648
25000 100000
20

Look at the first line: it is what Docker kept (NanoCpus counts in billionths of a core, hence 250,000,000 for a quarter of a core). Then look at the next three, read inside the container: memory.max gives the memory, pids.max the processes, and cpu.max holds two numbers, the computing time allowed (25,000 microseconds) in each period of 100,000. Both readings must say the same thing. If only the first one is right, Docker did not apply the limit.

A CPU limit can only be observed by loading the container. The cpu.stat file counts, in nr_throttled, the times the kernel held the container back because it exceeded its quota. Read this counter, run a busy loop for three seconds, then read it again.

docker compose exec indexeur sh -c 'grep nr_throttled /sys/fs/cgroup/cpu.stat; timeout 3 sh -c "while :; do :; done"; grep nr_throttled /sys/fs/cgroup/cpu.stat'

nr_throttled 0
Terminated
nr_throttled 31

Look at the difference between the two counters: it proves that the quota does slow the container down. The exact number changes at every attempt (0 then 31 here, 1 then 31 at the next attempt), it is the increase that counts. The word Terminated comes from timeout, which stopped the loop as planned.

What happens when a service reaches its process limit?

The kernel refuses to create the next process: the program that wanted to launch it receives an error, and nothing is killed. To see it, ask the container to launch thirty small processes in a row while it admits only twenty.

docker compose exec indexeur sh -c 'for i in $(seq 1 30); do sleep 5 & done; wait'

sh: can't fork: Resource temporarily unavailable

The message comes from the shell of the container, which could not create a new process and gave up its loop. The trap comes next: the container stays unusable. Seven seconds after the attempt, docker compose exec indexeur echo ok still failed, with OCI runtime exec failed ... unable to spawn stage-1: Resource temporarily unavailable. You can no longer open a shell in it until it is recreated, which you will do in a moment.

Keep this in mind: a process limit that is too low does not merely refuse a launch, it can also keep you from connecting to the container to understand what happened.

Optional: why the container stays blocked

The pids.current counter of the container was 19 for a limit of 20, and the host listed 18 sleep processes in the zombie state. A zombie is a process that has ended but that its parent has not yet reaped. Here the parent is process 1 of the container, sleep infinity, which reaps nothing. The zombies therefore stay counted in the limit until the container is recreated.

What happens to a limit set with docker update?

Before recreating the container, set a limit by hand with docker update, then read it. The two options --memory and --memory-swap receive the same value here.

docker update --memory 96m --memory-swap 96m budgets-indexeur-1
docker inspect -f 'memory={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}} processes={{.HostConfig.PidsLimit}}' budgets-indexeur-1

budgets-indexeur-1
memory=100663296 cpu=250000000 processes=20

Docker did take the new value (96 MiB is 100,663,296 bytes). The container has not answered exec since the previous attempt, so memory.max cannot be read again here; on a healthy container, it returned 100663296. Now recreate the container and read the limits again.

docker compose up -d --force-recreate indexeur
docker inspect -f 'memory={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}} processes={{.HostConfig.PidsLimit}}' budgets-indexeur-1

memory=50331648 cpu=250000000 processes=20

The limits written in compose.yaml are back, because Compose applies them at every creation: the 48 MiB replaced the 96 MiB set by hand. This recreation takes about ten seconds (10.4 s measured): sleep infinity ignores the stop request (it is process 1 of the container, with no signal handler, as the section on kill -9 1 explains), and Docker waits for the end of the grace period, explained in the section "Why can a stop take longer than expected?", before killing it.

A limit set with docker update does not survive a recreation, and the file does not contain it. If it must last, write it in compose.yaml.

What happens when a service exceeds its memory limit?

The kernel kills the process that consumes too much, and Docker records the cause in the state of the container: OOMKilled is then true. OOM stands for out of memory. Only the memory limit makes the kernel kill the service: exceeding the CPU only slows it down, and exceeding the processes makes the creation of the next one fail.

Add a service that consumes memory without ever stopping: the tail /dev/zero command reads an endless stream of zeros, with no line break, and keeps everything in memory. The service has a limit of 32 MiB, and a memswap_limit equal to mem_limit forbids using swap, the memory exchanged with the disk, to get around the limit. Its restart policy on-failure:2 is explained below.

  gourmand:
    image: alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
    command: ["tail", "/dev/zero"]
    mem_limit: 32m
    memswap_limit: 32m
    restart: "on-failure:2"

Start it, give it a few seconds, then read its state.

docker compose up -d gourmand
sleep 8
docker inspect -f 'status={{.State.Status}} code={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' budgets-gourmand-1

status=exited code=137 oom=true restarts=2

Look at oom=true: the container died for lack of memory, not because it was stopped. Code 137 alone would not be enough to say so, because it is also the code of a container killed by docker kill (the program is then killed by signal 9, and 128 + 9 = 137). OOMKilled=true signals it. Also note restarts=2: Docker restarted the service twice, then gave up. Running out of memory is therefore a failure like any other for the restart policy, as the next section shows.

How do you choose the restart policy of a service?

With the restart key of the service. It takes the same four values as the --restart option of docker run, detailed in the troubleshooting lesson cited above.

Value	Docker restarts the container…
"no" (default)	never
on-failure[:n]	when the program exits with a non-zero code, at most n times; when Docker restarts, only if the last exit was an error (137 included), never after an exit with 0
always	at every exit, including code 0; a container you had stopped starts again when Docker restarts
unless-stopped	like always, but a container you stopped yourself stays stopped, even when Docker restarts

Swipe to see more

For a service that must run permanently, unless-stopped is a good fit: it puts the service back on its feet after a crash or a restart of the machine, without restarting it if you stopped it on purpose. The demonstrations that follow use it.

Write "no" between quotes. With Compose 5.6.1, restart: no without quotes is accepted with no message at all: docker compose config displays it as restart: 'no' and docker inspect returns the policy no. Quotes remain the right habit, because in YAML 1.1 the word no means the boolean false, and other tools would read it that way.

What does a policy restart, and what does it leave stopped?

A policy restarts a process that dies by itself, and it does not touch a container that you stop or kill with a command. To see it, you therefore need two services: one that crashes by itself, and one that lives until you kill it. Add them to the file, both under unless-stopped.

  fragile:
    image: alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
    command: ["sh", "-c", "echo start; sleep 4; exit 1"]
    restart: unless-stopped

  veilleur:
    image: alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
    command: ["sleep", "infinity"]
    restart: unless-stopped
    stop_grace_period: 3s

The fragile service prints a message, waits four seconds, then exits with code 1. The veilleur service waits indefinitely; its stop_grace_period: 3s line is not used yet, the section "Why can a stop take longer than expected?" explains it. Start both, wait about fifteen seconds and read their counters.

docker compose up -d fragile veilleur
sleep 15
docker inspect -f '{{.Name}} status={{.State.Status}} restarts={{.RestartCount}}' budgets-fragile-1 budgets-veilleur-1

/budgets-fragile-1 status=running restarts=3
/budgets-veilleur-1 status=running restarts=0

Look at restarts for fragile: it is 3 here, and 2 or more in general, because each exit with code 1 was followed by a restart. The exact number depends on the moment of the reading. The status may also show restarting if you read between two attempts.

Now kill veilleur with docker kill, which sends SIGKILL immediately, then read its state again. The docker compose ps -q veilleur command gives the identifier of the container of the service, which docker kill expects.

docker kill $(docker compose ps -q veilleur)
sleep 3
docker inspect -f 'status={{.State.Status}} code={{.State.ExitCode}} restarts={{.RestartCount}}' budgets-veilleur-1

b0615a52e79a1e37cc2c7c694c31e85d5fd6c29ccbfcfeab2d6fd14d748986fc
status=exited code=137 restarts=0

The first line is the identifier that docker kill prints, it changes at every attempt. The container stayed stopped (exited), with code 137, and its counter did not move: docker kill does not trigger the restart policy at that moment, not even unless-stopped. Docker considers that you meant to stop it. The table compares the four ways of stopping a container in this lesson. The first row was measured under unless-stopped, the second under on-failure:2, and the last two under unless-stopped.

What stops the container	Restarted by the policy?
The program exits by itself with an error (fragile)	yes
The kernel kills it for lack of memory (gourmand)	yes, within the limit set by on-failure:n
docker stop or docker compose stop	no, at that moment
docker kill	no, at that moment

Swipe to see more

This "no" holds at that moment. When Docker restarts, always restarts a container stopped by hand, unless-stopped leaves it stopped, and on-failure restarts a container whose last exit was an error, hence also a killed container (code 137), but not a container that exited with 0. The detail is in the lesson Read a Docker startup failure.

Keep the consequence in mind: killing a container with docker kill does not simulate a crash. To test a restart policy, you need a program that crashes by itself, like fragile.

Can you simulate a crash by killing the main process from inside?

Not with kill -9 1. The main process of the container has number 1 in its own process space, and the kernel protects it from signals sent from that space: the command does nothing. Try it on veilleur, after starting it again.

docker compose start veilleur
docker compose exec veilleur kill -9 1
docker inspect -f 'status={{.State.Status}} restarts={{.RestartCount}}' budgets-veilleur-1

status=running restarts=0

The container is still running and the counter did not move: the signal was ignored. This special treatment of process 1 is the same one that makes it ignore SIGTERM, described in Container lifecycle, PID 1 and clean shutdown. Remember that this gesture, often suggested to "simulate a crash", proves nothing.

How do you restart a service by hand?

With docker compose restart <service> for a running service, and docker compose start <service> for a stopped service. The restart command stops the container then starts it again, respecting the grace period explained in the section "Why can a stop take longer than expected?". It does not reread the file: a change of limit or policy is only taken into account by docker compose up -d, which recreates the container.

How can you see that restart does not reread the file?

Take a setting that is easy to read, the grace period of veilleur. Change it in the file (from 3s to 5s), then read the delay the container kept after a restart, then after an up -d. Then put 3s back. The progress lines of Compose are not reproduced.

sed -i 's/stop_grace_period: 3s/stop_grace_period: 5s/' compose.yaml
docker compose restart veilleur
docker inspect -f 'delay={{.Config.StopTimeout}}' budgets-veilleur-1
docker compose up -d veilleur
docker inspect -f 'delay={{.Config.StopTimeout}}' budgets-veilleur-1
sed -i 's/stop_grace_period: 5s/stop_grace_period: 3s/' compose.yaml
docker compose up -d veilleur

delay=3
delay=5

After restart, the container keeps its old delay of 3 seconds; only up -d recreates it with the 5 seconds of the file. The last command puts the delay back to 3 seconds, a value the section "Why can a stop take longer than expected?" needs.

Why does docker compose ps -q return nothing after a stop?

Scripts find the identifier of a container with docker compose ps -q <service>. What this command returns depends on the state of the service: compare it with and without the -a option, before and after a stop.

docker compose restart veilleur
docker compose ps -q veilleur
docker compose stop veilleur
docker compose ps -q veilleur
docker compose ps -aq veilleur

 Container budgets-veilleur-1 Restarting
 Container budgets-veilleur-1 Started
537690deee963829b589a652f700437e7837fcb860dad631a78d988bc357272f
[+] stop 1/1
 ✔ Container budgets-veilleur-1 Stopped                                   3.1s
537690deee963829b589a652f700437e7837fcb860dad631a78d988bc357272f

Look at the identifiers. The first ps -q, run while the service is running, returns the identifier of the container. The second, run after stop, returns nothing: the -q option only lists the containers that are running, and the absence of a line is not an oversight. The -aq option adds the stopped containers and finds the identifier again. If the output of your script suddenly becomes empty, first check that the service is not simply stopped.

Why can a stop take longer than expected?

Because Docker gives the program a grace period before killing it. At a stop, it first sends an end signal (SIGTERM), a message that asks the program to stop, then it waits. If the container is still running at the end of the delay, it sends SIGKILL, a signal that the program cannot refuse. The default delay is ten seconds. The full mechanism, with the cases where the program ignores SIGTERM, is in the lesson Container lifecycle, PID 1 and clean shutdown.

In Compose, the stop_grace_period key sets this delay for a service. The veilleur service ignores SIGTERM: its main process, sleep, is process 1 of the container and has no handler for this signal. Its delay is set to 3 seconds. Measure the duration of a stop.

docker compose start veilleur
time docker compose stop veilleur
docker inspect -f 'code={{.State.ExitCode}}' budgets-veilleur-1

[+] stop 1/1
 ✔ Container budgets-veilleur-1 Stopped                                   3.2s

real    0m3.237s
code=137

Run start first: on a container that is already stopped, stop returns immediately (0.06 s measured) and proves nothing. Look at the duration: a little over three seconds, which is the delay that was set, not the ten of the default. Code 137 confirms that Docker had to send SIGKILL, for lack of an answer to SIGTERM. A delay that is too short exposes a service that needs to flush its data, such as a database, to being killed in the middle of its work. A delay that is too long slows down every stop and every redeployment.

What should you do when these settings do not behave as expected?

Start from what you see, then check the configuration that Docker really received before changing the file.

What you observe	Probable cause	What you do
docker compose config or up stops on additional properties '...' not allowed or must be a mapping	The key is at the wrong indentation level	Read the path of the message, put the key back under the service (or under deploy.resources.limits), then run docker compose config again
docker compose exec answers unable to spawn stage-1: Resource temporarily unavailable	The process limit is reached, zombies are still counted	Run docker compose up -d --force-recreate <service>, and raise the limit if it is too low
The container exits with code 137	Killed by SIGKILL: lack of memory, docker kill or grace period exceeded	Read .State.OOMKilled: true signals an OOM; false does not rule it out (a 137 without OOMKilled was measured once in five runs on Docker 29.9.2). Without OOMKilled or an oom event in docker events, you cannot conclude; consumption close to the limit, read before the stop, makes an OOM plausible, as shown by the lesson on memory and task ceilings
The container uses more than mem_limit without being killed, and OOMKilled stays false	Without memswap_limit, it can use swap on top of its memory, if the host has any	Add memswap_limit with the same value as mem_limit
A service killed with docker kill does not restart	Normal behavior: a stop command restarts nothing at that moment	Make the program itself crash to test the policy
kill -9 1 in the container has no effect	The kernel protects process number 1	Use a program that exits by itself; docker kill from the host kills the container but does not trigger the policy at that moment
docker compose ps -q returns nothing	The service is stopped, -q only lists the containers that are running	Add -a: docker compose ps -aq <service>
A change of restart or of a limit has no effect after docker compose restart	restart does not reread the file	Run docker compose up -d
The stop lasts ten seconds, then code 137	The program ignores SIGTERM, the default delay runs out	Fix the signal handling, or set stop_grace_period

Swipe to see more

How do you clean everything up after the demonstration?

The demonstration leaves four containers, one network, one image and the ~/budgets folder. The docker compose down command removes the containers and the network, including those that are stopped.

cd ~/budgets
docker compose down
docker rmi alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
cd ~ && rm -r budgets

After that, docker ps -a shows no budgets-... container, and ls ~/budgets answers that the folder no longer exists.

Practice

The lab gives you a host with Docker and the pinned Alpine image, with no load at all. You write the compose.yaml of a single worker service that waits, with memory, CPU and process limits whose values the statement sets. The service must have no mount, no extra capability and no privilege, and the project name is imposed with the -p option. The CPU must be a hard quota, not a relative weight.

The validation compares what Docker recorded (HostConfig) with what the kernel enforces in the container (memory.max, cpu.max, pids.max). It then runs a busy loop of three seconds and requires the CPU throttling counter to increase, then it forcibly recreates the service and reads the three limits again. It checks neither the lack of memory nor a crash: these two cases remain demonstrations of the lesson.

Guided lab Practice

Learn how to do it

Enforce Compose resource budgets on the engine

$ dsoxlab run compose-ressources

Environment: VM·Duration: ~30 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start compose-ressources checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/containers-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run compose-ressources
dsoxlab check compose-ressources

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

1. The mem_limit, cpus and pids_limit keys set hard ceilings on a service. docker compose config shows what Docker will receive, and refuses with a message a limit written at the wrong level.

2. A limit is checked in two places: docker inspect (what Docker recorded) and the memory.max, cpu.max and pids.max files in the container (what the kernel enforces). The CPU limit is proven by the increase of nr_throttled in cpu.stat.

3. Only the memory limit makes the kernel kill the service. OOMKilled set to true signals an OOM; with false, it is not ruled out.

4. A process limit that is too low kills nothing, but it can make the container unusable (docker compose exec fails) until it is recreated.

5. The limits written in compose.yaml survive a recreation. A limit set with docker update does not.

6. The restart key restarts a program that dies by itself. docker stop and docker kill do not trigger it at that moment (when Docker restarts, the result depends on the policy), and kill -9 1 from the inside does nothing: only a program that crashes by itself demonstrates a policy.

7. docker compose ps -q only lists the services that are running, -aq includes the stopped ones. docker compose restart does not reread the file, docker compose up -d does.

8. At a stop, Docker sends SIGTERM then, after stop_grace_period (ten seconds by default), SIGKILL. Set this delay according to the time the service needs to stop cleanly.

Knowledge check

Check that the essentials of this lesson have stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

8 questions

8 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →