Understanding Linux cgroups

A process goes into a loop and saturates a core. A service leaks and eats all the RAM. Cgroups (control groups) are the Linux kernel mechanism that prevents this: they group processes and cap what that group may consume in CPU, memory, disk input/output and number of processes. This page shows you how to set these limits, how to prove they apply, and above all why, on a modern server, the right interface is not /sys/fs/cgroup but systemd. The examples run on Ubuntu 24.04 (kernel 6.8), but the behaviour is the same on Debian 13 and RHEL 10, which all use cgroups v2 by default.

What you will learn

Check in one command whether your machine runs cgroups v1 or v2.
Create a cgroup, put a process in it and limit it to 20% of a core, with proof.
Cap memory and watch the OOM kill triggered by the cgroup.
Throttle disk I/O and the number of processes (pids.max).
Limit a service with systemctl set-property, the only method that survives a reboot.
Understand what Docker and Kubernetes really do when you write --memory=128m.

Which cgroups version are you on

Two generations of cgroups coexist in the kernel and they have neither the same files nor the same paths. Mixing them up is the first source of commands that do not work: a tutorial that has you write to /sys/fs/cgroup/cpu/cpu.cfs_quota_us is talking about v1, which is no longer the default mode of any mainstream distribution. It has not disappeared from the kernel, though: on a Debian 13 with kernel 6.12, a mount -t cgroup -o cpu still mounts a complete v1 hierarchy, and systemd.unified_cgroup_hierarchy=0 brings it back at boot. You can therefore meet it on an old machine or one that was deliberately switched back. The command that settles it queries the type of the mounted file system.

stat -fc %T /sys/fs/cgroup

cgroup2fs

The cgroup2fs answer means unified cgroups v2: a single hierarchy, every controller in the same place. A tmpfs answer would mean v1 or hybrid mode. This whole page assumes v2, which is the default on Ubuntu 22.04+, Debian 11+ and RHEL 9+.

A controller (cpu, memory, io, pids…) is the kernel module that enforces one category of limit. The root tells you which ones are compiled in and available.

cat /sys/fs/cgroup/cgroup.controllers

cpuset cpu io memory hugetlb pids rdma misc

Limit a process to 20% CPU

The best way to understand cgroups is to build one and watch it bite. A cgroup is a plain directory under /sys/fs/cgroup: creating it is enough to make it exist, and the kernel immediately drops the control files in it. All these operations need root rights, and since the > redirection is run by your shell (not by sudo), you must go through sudo tee to write to these files.

sudo mkdir /sys/fs/cgroup/demo
cat /sys/fs/cgroup/demo/cgroup.controllers

cpuset cpu io memory hugetlb pids rdma misc

These controllers are usable here because the parent (the root) has delegated them to its children. The limit files are therefore already present, in the open position.

The exact list depends on the kernel: read it rather than assume it. The one above comes from Ubuntu 24.04; on AlmaLinux 10.2 (kernel 6.12.0-211.el10_2), the same command returns one more controller, dmem, which rations device memory (that of a graphics card, for example). The kernel documentation's cgroup-v2 page now describes it in its DMEM section. The rest of this page behaves identically on both distributions, checked command by command.

cat /sys/fs/cgroup/demo/cpu.max
cat /sys/fs/cgroup/demo/memory.max
cat /sys/fs/cgroup/demo/pids.max

max 100000
max
max

Now let us start an infinite loop that saturates a core, then attach it to the cgroup. The cgroup.procs file holds the list of the group's PIDs (process identifiers): writing a PID to it moves the process, which immediately inherits the limits in force.

while :; do :; done &
# PID=718500
echo 718500 | sudo tee /sys/fs/cgroup/demo/cgroup.procs
cat /proc/718500/cgroup

0::/demo

The 0::/demo line confirms the attachment. A process belongs to a single cgroup at a time: putting it in one automatically removes it from the previous one. Without a limit, it consumes 100% of a core, as top confirms.

top -b -n 1 -p 718500 | tail -1

 718500 bob  20  0  7348  1808  1512 R 100,0  0,0  0:05.23 bash

The cpu.max file expects two numbers: a quota and a period, in microseconds. Writing 20000 100000 allows 20 ms of CPU time per 100 ms slice, that is 20% of a core. The limit applies immediately to the processes already attached, with no restart and no signal.

echo "20000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max
top -b -n 1 -p 718500 | tail -1

 718500 bob  20  0  7348  1808  1512 R  18,2  0,0  0:06.69 bash

The process dropped from 100% to 18%, without being changed or restarted. The cpu.stat file tells the story from the kernel side: it counts the periods during which the group was throttled, that is paused because it had used up its quota.

cat /sys/fs/cgroup/demo/cpu.stat

usage_usec 6677619
user_usec 6676621
system_usec 997
nr_periods 72
nr_throttled 72
throttled_usec 5719555

nr_throttled 72 out of nr_periods 72: every period was throttled. It is the signature of a process that hits its ceiling all the time. On a production service, a rising nr_throttled is the symptom of a quota that is too tight, not of an overloaded machine: it is the metric to watch before blaming the hardware.

The Ubuntu 6.8 kernel adds three lines to this file, removed from the output above for readability: core_sched.force_idle_usec, present when the kernel is built with core scheduling, then nr_bursts and burst_usec, which count the overruns allowed by cpu.max.burst. They stay at 0 as long as you have configured nothing of the kind.

The rule that breaks most tutorials

The cgroup.subtree_control file is the number one trap of cgroups v2, and it deserves a pause before going further. Many guides have you write +cpu +memory to it just before putting a process in. That sequence always fails, and here is why.

cgroup.subtree_control does not enable the controllers for the cgroup you write to, but for its future sub-groups. And the kernel enforces the so-called "no internal processes" rule: a cgroup that delegates controllers to its children can no longer contain processes. It becomes a mere organisational node. On an empty cgroup, it is adding the process that fails:

echo "+cpu +memory" | sudo tee /sys/fs/cgroup/demo/cgroup.subtree_control
echo 721148 | sudo tee /sys/fs/cgroup/demo/cgroup.procs

tee: /sys/fs/cgroup/demo/cgroup.procs: Device or resource busy

On a cgroup that already contains processes, such as demo with the loop from the previous section, it is the write to cgroup.subtree_control itself that is refused, with the same Device or resource busy. Both cases were measured on kernel 6.8.

To limit processes, do not touch the subtree_control of the cgroup that will host them: its limit files are already there, inherited from the parent. subtree_control is only used to build a hierarchy of nested groups.

To go back, remove the controllers with the - prefix, and the cgroup can host processes again. This only works as long as no sub-group uses them.

echo "-cpu -memory" | sudo tee /sys/fs/cgroup/demo/cgroup.subtree_control

Cap memory and trigger the OOM kill

The memory limit is the most brutal of all, because an allocation cannot be "slowed down": when a process asks for a page and there is no quota left, the kernel kills someone in the cgroup. The memory.max file sets that ceiling and it accepts the suffixes K, M, G, which saves you counting bytes by hand. Reading it back, however, always returns bytes.

echo "512M" | sudo tee /sys/fs/cgroup/demo/memory.max
cat /sys/fs/cgroup/demo/memory.max

536870912

To watch it trigger, let us lower the ceiling to 64 MB and turn off the group's swap (otherwise the kernel just buys time by swapping instead of killing). A small Python program then allocates memory one megabyte at a time, launched by a script that first attaches itself to the cgroup.

echo "64M" | sudo tee /sys/fs/cgroup/demo/memory.max
echo "0" | sudo tee /sys/fs/cgroup/demo/memory.swap.max

The two files below are enough to reproduce the experiment. The script writes its own PID to cgroup.procs, then exec replaces the shell with Python without changing PID: it is therefore Python that inherits the ceiling. Run it as root, since it writes to /sys/fs/cgroup.

# oom.py: allocates 1 MB at a time and announces it
blocs = []
for i in range(1, 200):
    blocs.append(bytearray(1024 * 1024))
    print(f"{i} Mo alloues", flush=True)

#!/bin/bash
# oom.sh: attaches itself to the demo cgroup, then becomes python3
echo $$ > /sys/fs/cgroup/demo/cgroup.procs
exec python3 ./oom.py

The process is attached to the cgroup before it starts allocating, then it counts out loud.

58 Mo alloues
59 Mo alloues
60 Mo alloues
bash: 719360 Killed                  ./oom.sh
code de sortie : 137

The process dies at 60 MB, killed by the kernel, with exit code 137 (128 + signal 9). That number is exactly the one you will see on an OOMKilled container. The memory.events file keeps a record of the incident, and it is the one to read for an after-the-fact diagnosis.

cat /sys/fs/cgroup/demo/memory.events

low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0

Three counters really matter. max 35 means the group hit its ceiling 35 times, forcing the kernel to reclaim memory: a max that climbs without oom_kill reveals a cgroup that is too tight, spending its time recycling pages instead of working. oom 1 says an allocation could not be satisfied, and oom_kill 1 that a process was shot down for it. Replayed in a privileged container on the same kernel, the experiment returns exactly these six counters, max 35 included. The kernel log names the culprit and the context.

sudo dmesg | tail -2

oom-kill:constraint=CONSTRAINT_MEMCG,oom_memcg=/demo,task_memcg=/demo,task=python3,pid=719360
Memory cgroup out of memory: Killed process 719360 (python3) total-vm:80560kB, anon-rss:65152kB

CONSTRAINT_MEMCG is the keyword that tells a cgroup OOM (a single application exceeds its quota, the rest of the machine is fine) from a system OOM (the whole machine is out of RAM). Mixing the two up sends you looking for the problem in the wrong place for hours.

Throttle disk I/O and the number of processes

The io controller caps disk throughput through the io.max file, which expects the block device's major:minor number followed by the desired limits: rbps and wbps for bytes per second when reading and writing, riops and wiops for the number of operations. These numbers come from lsblk.

lsblk -o NAME,MAJ:MIN,TYPE

NAME                      MAJ:MIN TYPE
nvme0n1                   259:0   disk
├─nvme0n1p3               259:3   part
  └─ubuntu--vg-ubuntu--lv 252:0   lvm

The trap lies here, and it is subtler than you often read. A limit set on a major:minor that is not on the I/O path actually taken is written without error and slows nothing down, which wastes a considerable amount of time. On the other hand, there is no absolute rule saying you should always target the physical disk rather than the logical volume.

Measured on an LVM on loop device stack, reading 20 MB in direct I/O, limit at 1 MB/s: the table compares the duration of the same read depending on the layer targeted.

Target of the limit	Duration
none	16 ms
the logical volume (`254:0`)	20,005 ms
the underlying device (`7:0`)	19,996 ms

Both layers throttle, and indistinguishably so. With device-mapper, accounting and throttling can apply at several levels depending on the I/O path: the right method is therefore to identify the major:minor of the layer you want to throttle, then to check by measurement that the limit bites. Do not assume, time it.

So let us limit reads to 1 MB/s, then reread a file in direct I/O (iflag=direct) to bypass the page cache, otherwise the read would be served from RAM and would never touch the disk. It is the second, very frequent, cause of a limit that seems to do nothing.

echo "259:0 rbps=1048576" | sudo tee /sys/fs/cgroup/demo/io.max
dd if=fichier-test of=/dev/null bs=1M count=20 iflag=direct

20971520 bytes (21 MB, 20 MiB) copied, 19,9714 s, 1,1 MB/s

The same read reached 1.2 GB/s without a limit: the ceiling is respected to the percent. The pids controller plays in another league: it does not ration a hardware resource but the number of processes, which makes it the simplest barrier against a fork bomb or an application leaking threads. The pids.max file takes an integer.

echo 5 | sudo tee /sys/fs/cgroup/demo/pids.max

Any process creation beyond the fifth is then refused by the kernel, and the failure goes all the way up to the shell. The test script attaches itself to the cgroup, then starts seven sleep processes in the background:

#!/bin/bash
# pids.sh: seven processes for a ceiling of five
echo $$ > /sys/fs/cgroup/demo/cgroup.procs
for i in 1 2 3 4 5 6 7; do sleep 30 & done
wait

./pids.sh: fork: retry: Resource temporarily unavailable
./pids.sh: fork: Resource temporarily unavailable

The pids.events counter records the refusals (max 5 here), which lets you spot an application that brushes against its limit before it breaks.

What cgroups cannot do

A very widespread belief holds that cgroups also limit network bandwidth. It is false, and the confusion comes from v1. The old net_cls and net_prio controllers merely marked or prioritised packets; they capped no throughput, and they have no equivalent in cgroups v2. The file you are looking for does not exist.

To throttle network throughput, you must step outside cgroups and use tc (traffic control), the kernel's traffic shaping subsystem, or an eBPF program attached to the cgroup, which is what Cilium does on Kubernetes. Remember the boundary: cgroups manage CPU, memory, block I/O, PIDs and devices, and stop there.

systemd is the real owner of cgroups

This is the point most tutorials leave out, and it is the most important one on this page. On every modern distribution, systemd is the machine's cgroup manager. It claims exclusive ownership, puts each service in its own cgroup and reapplies its values whenever it gets the chance. Everything you wrote by hand in /sys/fs/cgroup for a service managed by systemd is therefore temporary.

The demonstration leaves no doubt, and it was replayed with Ubuntu 24.04's systemd 255. Take a service started by systemd-run with a 20% CPU quota, overwrite cpu.max directly in /sys, then run a simple daemon-reload.

sudo systemd-run --unit=demo-burn -p CPUQuota=20% /bin/bash -c 'while :; do :; done'
cat /sys/fs/cgroup/system.slice/demo-burn.service/cpu.max
echo "80000 100000" | sudo tee /sys/fs/cgroup/system.slice/demo-burn.service/cpu.max
sudo systemctl daemon-reload
cat /sys/fs/cgroup/system.slice/demo-burn.service/cpu.max

20000 100000
80000 100000
20000 100000

Your change lived a few seconds. A daemon-reload, a set-property on any other property, a restart of the service or of the machine, and systemd restores its view of the world. Add to that the fact that /sys/fs/cgroup is a virtual kernel file system, recreated empty at every boot: your mkdir never persisted anything.

Limit a service properly

The systemctl set-property command applies the limit immediately and writes it to a configuration file that survives a reboot. It accepts the same directives as unit files.

sudo systemctl set-property demo-burn.service CPUQuota=20% MemoryMax=256M

Systemd translates these directives into writes in the service's cgroup, exactly the ones you would have done by hand, except that it now remembers them: that is what protects them from the next daemon-reload.

cat /sys/fs/cgroup/system.slice/demo-burn.service/cpu.max
cat /sys/fs/cgroup/system.slice/demo-burn.service/memory.max

20000 100000
268435456

Persistence is provided by a drop-in, a configuration fragment that extends the unit without changing it. For an installed service, such as atd, systemd puts it under /etc/systemd/system.control/, the directory dedicated to properties set from the command line. For a transient unit such as demo-burn, created by systemd-run, it lands under /run/systemd/transient/, and disappears with it. The file below was produced by a sudo systemctl set-property atd.service CPUQuota=10%.

cat /etc/systemd/system.control/atd.service.d/50-CPUQuota.conf

# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Service]
CPUQuota=10%

For a setting you want to version with the rest of your configuration, prefer a drop-in written by hand through sudo systemctl edit name.service, which holds the same directives. CPUQuota=20% maps to cpu.max, MemoryMax=256M to memory.max, TasksMax=100 to pids.max, and IOReadBandwidthMax=/dev/nvme0n1 5M to io.max. To cleanly undo a property set by set-property, sudo systemctl revert name.service removes the drop-in and puts the service back in its original state.

Check what is really applied

Mind the property names: systemctl show name.service -p CPUQuota returns nothing. CPUQuota is a unit directive, not a property exposed by systemd. The property to query has another name and is expressed in CPU time per second.

systemctl show demo-burn.service -p CPUQuotaPerSecUSec -p MemoryMax

CPUQuotaPerSecUSec=200ms
MemoryMax=268435456

200ms per second is indeed the 20% requested. The systemctl status command gives the same information in a more readable form, showing the active drop-in, the current consumption and the service's cgroup.

● demo-burn.service - Demo cgroups
    Drop-In: /run/systemd/transient/demo-burn.service.d
             └─50-CPUQuota.conf, 50-MemoryMax.conf
     Memory: 344.0K (max: 256.0M available: 255.6M peak: 848.0K)
     CGroup: /system.slice/demo-burn.service
             └─722975 /bin/bash -c "while :; do :; done"

Explore the cgroup tree

Systemd provides two commands that advantageously replace exploring with cat in /sys. systemd-cgls shows the tree of cgroups with the processes they contain, which instantly answers the question "which service does this process belong to".

systemd-cgls --no-pager -u system.slice

Unit system.slice (/system.slice):
├─containerd.service …
│ ├─  1287 /usr/bin/containerd
│ ├─  2335 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 259166de4274f28…
├─systemd-networkd.service
│ └─603 /usr/lib/systemd/systemd-networkd

systemd-cgtop is its dynamic counterpart: a top per cgroup, which sorts services by consumption. It is the tool that answers in seconds the question "which service is eating this server's CPU", where top only shows isolated processes without saying who they belong to.

systemd-cgtop -b -n 1 --depth=2

CGroup                     Tasks   %CPU   Memory  Input/s Output/s
/                           2469      -    11.5G        -        -
system.slice                1146      -    10.2G        -        -
system.slice/atd.service       1      -   504.0K        -        -
init.scope                     1      -     9.9M        -        -

What Docker and Kubernetes really do

When you write docker run --cpus=0.5 --memory=128m, Docker does nothing other than what you have just done by hand. It creates a cgroup for the container and writes your values to it. The proof takes three commands, replayed with Docker 29.1. The engine first announces that it delegates to systemd and works in v2.

docker info | grep -i cgroup

 Cgroup Driver: systemd
 Cgroup Version: 2

Let us start a limited container, then read its cgroup on the host. The cgroup name contains the container's full ID, in a scope filed under system.slice.

docker run -d --rm --name demo-cg --cpus=0.5 --memory=128m alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8 sleep 120
cat /sys/fs/cgroup/system.slice/docker-8920e22899cd....scope/cpu.max
cat /sys/fs/cgroup/system.slice/docker-8920e22899cd....scope/memory.max

50000 100000
134217728

--cpus=0.5 became 50000 100000, and --memory=128m became 134217728 bytes. Nothing magic: a container is a set of processes in a cgroup (for resources) and in namespaces (for isolating the view of the system). Kubernetes applies the same mechanism one notch higher, but not symmetrically for the two resources, and that is the shortcut to undo. Recorded on the node, in the Pod's cgroup, with Kubernetes 1.37:

What the Pod declares	What you read in its cgroup
`requests.cpu: 100m`	`cpu.weight: 4`, a weight
`limits.cpu: 500m`	`cpu.max: 50000 100000`, a ceiling
`limits.memory: 300Mi`	`memory.max: 314572800`, a ceiling
`requests.memory: 200Mi`	nothing. `memory.max` is `max`, `memory.min` and `memory.low` are 0

There is no such thing as a "memory weight". A memory request produces no cgroup control: it helps the scheduler pick a node that has room, and ranks the Pod for eviction. Nothing on the node reserves those 200 Mi or protects them. That is why a container that exceeds its limits.memory ends up OOMKilled with code 137: it is the same cgroup OOM kill as the one observed above, with the same CONSTRAINT_MEMCG in dmesg.

Troubleshooting

Cgroups almost always fail silently or with misleading messages. Here are the errors actually met while building this page, with their cause and their fix.

Symptom	Cause	Solution
`mkdir: cannot create directory '/sys/fs/cgroup/demo': Permission denied`	`/sys/fs/cgroup` belongs to root	Prefix with `sudo`
`Read-only file system` on every write	You are in a container: Docker mounts `/sys/fs/cgroup` read-only	Redo the exercise on a VM or on the host
`bash: /sys/fs/cgroup/demo/cpu.max: Permission denied` with `sudo echo ... >`	The `>` redirection is done by your shell, not by `sudo`	Use `echo "..." \| sudo tee /sys/fs/cgroup/demo/cpu.max`
`tee: /sys/fs/cgroup/demo/cgroup.procs: Device or resource busy`	Controllers are enabled in the same cgroup's `cgroup.subtree_control` ("no internal processes" rule)	Remove them (`echo "-cpu -memory" \| sudo tee .../cgroup.subtree_control`) or put the processes in a sub-group
`tee: …/cgroup.subtree_control: Device or resource busy`	The cgroup already contains processes, same rule	Move the processes into a sub-group before delegating
`rmdir: failed to remove '/sys/fs/cgroup/demo': Device or resource busy`	The cgroup still contains processes	Empty `cgroup.procs` (stop or move the processes) before the `rmdir`
`io.max` written, but no slowdown	`major:minor` off the I/O path taken, or read served by the page cache	Read the numbers with `lsblk -o NAME,MAJ:MIN`, and time it with `dd iflag=direct`: the measurement settles it, not the layer chosen
`fork: Resource temporarily unavailable`	`pids.max` reached	Raise `pids.max` or fix the process leak
The value written to `/sys` goes back to its old value	systemd reapplies the unit's properties on `daemon-reload`, restart and boot	Go through `systemctl set-property` or a drop-in
The cgroup created by hand disappeared after a reboot	`/sys/fs/cgroup` is a kernel file system, recreated empty at boot	Declare the limit in a systemd unit
`systemctl show svc -p CPUQuota` returns nothing	`CPUQuota` is a unit directive, not an exposed property	Read `CPUQuotaPerSecUSec`

Where cgroups come from

A little context helps understand why two versions coexist. Cgroups entered the Linux kernel in 2007, pushed by Google engineers who had to make thousands of applications live together on the same servers. That v1 gave each controller its own independent hierarchy, which led to inconsistent trees and rules impossible to reason about globally.

v2, stabilised in 2016, brought everything back under a single hierarchy mounted on /sys/fs/cgroup, with consistent file names (cpu.max, memory.max, io.max). It is now the default mode everywhere, which does not mean the only possible one: the kernel can still mount v1, and a hybrid mode still exists on some fleets. The practical consequence is simple: any documentation older than 2019 probably talks about v1, and its paths no longer exist on your machine. Check with stat -fc %T /sys/fs/cgroup before following a tutorial.

Key points

stat -fc %T /sys/fs/cgroup must return cgroup2fs; otherwise you are on v1 and this page's paths do not apply.
A cgroup is a directory; you put a process in it by writing its PID to cgroup.procs, and the limits apply at once.
cgroup.subtree_control configures the children, never the cgroup itself. A cgroup that delegates controllers can no longer contain processes (Device or resource busy).
cpu.max throttles (nr_throttled in cpu.stat), memory.max kills (oom_kill in memory.events, exit code 137), io.max slows down, pids.max refuses fork calls.
cgroups v2 limits no network throughput: net_cls and net_prio only marked traffic. Use tc or eBPF.
systemd owns the cgroups: any echo into /sys/fs/cgroup for a service is overwritten at the first daemon-reload and lost at reboot.
The right method is systemctl set-property name.service CPUQuota=20%, checked with CPUQuotaPerSecUSec (not CPUQuota, which returns nothing).
systemd-cgls and systemd-cgtop are the exploration tools; Docker and Kubernetes only write to these same files.
A Kubernetes memory request becomes no cgroup setting. Recorded on the node: with a 200 Mi request and no limit, memory.max is max and both memory.min and memory.low are 0. Only the CPU request becomes a weight (cpu.weight). The requests/limits symmetry only holds for CPU.
No absolute rule on io.max and LVM: measured, the logical volume and the underlying device both throttle. What matters is being on the I/O path taken, and checking it by timing.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

The timer starts when you click Start
Multiple choice, true/false and short answer questions
You can move between questions
Detailed results are shown at the end

Understanding Linux cgroups

What you will learn

Which cgroups version are you on

Limit a process to 20% CPU

The rule that breaks most tutorials

Cap memory and trigger the OOM kill

Throttle disk I/O and the number of processes

What cgroups cannot do

systemd is the real owner of cgroups

Limit a service properly

Check what is really applied

Explore the cgroup tree

What Docker and Kubernetes really do

Troubleshooting

Where cgroups come from

Key points

Knowledge check

Knowledge check

How it works

Review

Score by domain

Summary

Strengths

Weak areas

Recommended next