Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes requests and limits: avoid the pitfalls

By updated

Improve this page

Logo Kubernetes

A Pod killed with the OOMKilled message, another one stuck in Pending for lack of room, a third one that slows down without ever restarting: in all three cases, the cause is to be found in the requests and the limits. A request is the amount of CPU or memory that the scheduler reserves for the container in order to place it on a node. A limit is the ceiling that the container cannot exceed. This lesson teaches you to set these two values and read them back, to trigger then fix an OOMKilled, to resize a Pod without restarting it, and to understand why a Pod stays Pending.

To follow this lesson, you need to know Pods and have the demo cluster created in First cluster. The kubectl top command, which displays the real consumption, needs metrics-server, the component that collects these measurements. It is presented in Read the cluster state: kubectl get, describe, logs and top, and you do not need it for the demonstration.

What you will learn

  • Write requests and limits and read them back with kubectl describe
  • Read the QoS class of a Pod (Guaranteed, Burstable or BestEffort) and understand what determines it
  • Recognize an OOMKilled (code 137) and fix it by raising the memory limit
  • Read the applied limit from inside the container, in the cgroup
  • Resize a Pod in place with --subresource resize
  • Diagnose a Pending Pod caused by insufficient resources

Understand requests and limits

Both fields are set per container, in resources, for the CPU and the memory. The CPU is counted in millicores (100m is a tenth of a core) and the memory in bytes (64Mi).

Requests: the guaranteed minimum

The request is used for placement: the scheduler only places a Pod on a node that still has enough unreserved resources. It is also a guarantee: a container gets at least its request, even when the node is loaded.

Limits: the barrier not to cross

The limit is a ceiling enforced by the kernel, and it does not behave the same way for the two resources. Above its CPU limit, a container is slowed down: this is throttling. Above its memory limit, it is killed: this is the OOMKilled (Out Of Memory) status, with exit code 137.

CPU resources of a Pod: the 250m request is guaranteed by the scheduler, the burst zone above it stays usable if the node has capacity, the 500m limit sets the ceiling, and beyond it come CPU throttling or OOMKilled.Request: 250mguaranteed by the schedulerBurst zoneusable if the node has capacityLimit: 500mceilingBeyondCPU throttling or OOMKilled
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Look at the three zones of the diagram: below the request, the resource is guaranteed; between the request and the limit, the container can expand if the node has capacity; above the limit, the CPU is slowed down and the memory is killed.

CPU throttling: when Kubernetes slows you down

A CPU limit of 500m grants 50,000 microseconds of CPU per 100,000-microsecond period: a container that consumes them before the end of the period waits for the next one. It is not killed, it is slowed down, and nothing shows up in the Pod's events. A slow Pod without any restart is the symptom to connect with a CPU limit that is too low, which the section on the cgroup lets you read back.

Read a Pod's resources

You work in a dedicated namespace, deleted at the end:

Terminal window
kubectl config use-context kind-demo && kubectl create namespace ressources
Switched to context "kind-demo".
namespace/ressources created

The names and messages of this lesson are in French on purpose, so that the outputs stay those of the tested cluster: ressources means "resources", garanti "guaranteed", extensible "expandable", libre "free", gourmand "greedy", trop-gros "too big", donnees "data", and the NOM column "NAME". The -corrige suffix of a file means "fixed". The echo messages "lecture de /dev/zero en mémoire" and "48 Mi en mémoire" mean "reading /dev/zero into memory" and "48 Mi in memory".

The qos.yaml file declares three nginx Pods: garanti has requests equal to the limits for the CPU and the memory, extensible has requests lower than a memory limit and no CPU limit, libre has no resources declared.

Terminal window
mkdir -p ~/atelier-k8s/ressources && cd ~/atelier-k8s/ressources
cat > qos.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: garanti
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 100m
memory: 64Mi
---
apiVersion: v1
kind: Pod
metadata:
name: extensible
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
memory: 128Mi
---
apiVersion: v1
kind: Pod
metadata:
name: libre
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
EOF
kubectl apply -f qos.yaml -n ressources && kubectl wait --for=condition=Ready pod --all -n ressources --timeout=90s
pod/garanti created
pod/extensible created
pod/libre created
pod/extensible condition met
pod/garanti condition met
pod/libre condition met

The three Pods are ready. Kubernetes has put each one in a quality of service class (QoS), which it computes from the Pod's resources. Read it:

Terminal window
kubectl get pods -n ressources -o custom-columns=NOM:.metadata.name,QOS:.status.qosClass
NOM QOS
extensible Burstable
garanti Guaranteed
libre BestEffort

The three QoS classes

The three classes are explained by the resources of your three Pods:

  • Guaranteed: each container has, for the CPU and for the memory, a request and a limit that are equal and greater than zero. This is the case of garanti.
  • Burstable: at least one request or one limit exists, without the Pod being Guaranteed. This is the case of extensible, which has no CPU limit and no request equal to its memory limit.
  • BestEffort: no request and no limit at all. This is the case of libre.

The class is not a setting: it is deduced from the resources that you declare. Now read the resources of the garanti Pod:

Terminal window
kubectl describe pod garanti -n ressources | grep -A8 "Limits:"
Limits:
cpu: 100m
memory: 64Mi
Requests:
cpu: 100m
memory: 64Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ml6ct (ro)

The Limits and Requests blocks are identical: this is what gives the Guaranteed class.

The class also tells how exposed a Pod is to eviction, but it is not the only criterion. When the node runs out of memory, the kubelet (the Kubernetes agent that runs on each node) evicts Pods, that is, it stops them to free room. It first picks the ones whose usage exceeds their requests, then it takes the Pod's priority into account. A BestEffort Pod, whose request is zero, always exceeds its request: it is therefore always among the eviction candidates.

OOMKilled: when your Pod blows up

The gourmand Pod reads /dev/zero into memory with tail, under a 32 Mi limit: it necessarily exceeds its limit.

Terminal window
cat > gourmand.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: gourmand
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command:
- sh
- -c
- |
echo 'lecture de /dev/zero en mémoire'
tail /dev/zero
resources:
requests:
cpu: 100m
memory: 32Mi
limits:
cpu: 500m
memory: 32Mi # ❌ trop bas pour ce que le conteneur lit en mémoire
EOF
kubectl apply -f gourmand.yaml -n ressources
pod/gourmand created

The comment in the manifest means "too low for what the container reads into memory". The container needs about twenty seconds to be killed, restarted and killed again. Wait 25 seconds before reading the state of the Pod, then the reason for the last termination of the container, which its status keeps:

Terminal window
sleep 25 && kubectl get pod gourmand -n ressources && kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
NAME READY STATUS RESTARTS AGE
gourmand 0/1 OOMKilled 2 (20s ago) 25s
OOMKilled 137

The OOMKilled status and the restarts that pile up show that the container is killed, then restarted. The last line is OOMKilled 137: the kernel killed the container because it exceeded its memory limit, and 137 is 128 plus signal 9. kubectl describe pod gourmand -n ressources shows the same thing, with Reason: OOMKilled and Exit Code: 137 in the Last State section, then Restart Count: 2.

Fix the limit

kubectl apply does not change the limits of an existing Pod: here, delete the Pod and recreate it with a suitable limit (another way exists, see "Resize a Pod in place"). The gourmand-corrige.yaml file writes 48 MB into /dev/shm, a directory that lives in memory, under a 128 Mi limit. It also adds resizePolicy, which is used further on for the in-place resize.

Terminal window
cat > gourmand-corrige.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: gourmand
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command:
- sh
- -c
- |
dd if=/dev/zero of=/dev/shm/donnees bs=1M count=48
echo "48 Mi en mémoire"
sleep 3600
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: NotRequired
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
EOF
kubectl delete pod gourmand -n ressources --wait=true && kubectl apply -f gourmand-corrige.yaml -n ressources && kubectl wait --for=condition=Ready pod/gourmand -n ressources --timeout=90s && sleep 5 && kubectl get pod gourmand -n ressources
pod "gourmand" deleted from ressources namespace
pod/gourmand created
pod/gourmand condition met
NAME READY STATUS RESTARTS AGE
gourmand 1/1 Running 0 6s

The Pod is Running with 0 restarts: 48 MB fit under 128 Mi.

Read the limit from inside the container

The limit set in the manifest becomes a kernel setting, visible in the container's cgroup (the Linux control group that enforces the limit). On a node using cgroup v2, two files give it:

Terminal window
kubectl exec gourmand -n ressources -- sh -c 'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/cpu.max'
134217728
50000 100000

134217728 is 128 Mi in bytes. 50000 100000 means 50,000 microseconds of CPU per 100,000-microsecond period: the 500m limit, that is half a core. Reading these files proves that the limit of the manifest is the one the kernel applies.

Resize a Pod in place

Until now, changing a resource meant recreating the Pod. Kubernetes 1.37 lets you resize a running container, through the resize subresource of the Pod. The resizePolicy field set above says that neither the CPU nor the memory requires a restart (NotRequired). The change goes through kubectl patch ... --subresource resize: the same patch without the subresource is refused, because a Pod only lets you change a few fields.

Raise the limits to 1 CPU and 256 Mi:

Terminal window
kubectl patch pod gourmand -n ressources --subresource resize --patch '{"spec":{"containers":[{"name":"app","resources":{"limits":{"cpu":"1","memory":"256Mi"}}}]}}'
pod/gourmand patched

The command is accepted. Wait for the kernel to apply the new limit, then read it back, with the number of restarts:

Terminal window
for i in $(seq 1 20); do [ "$(kubectl exec gourmand -n ressources -- cat /sys/fs/cgroup/memory.max)" = 268435456 ] && break; sleep 2; done; kubectl exec gourmand -n ressources -- sh -c 'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/cpu.max'; kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'
268435456
100000 100000
0

memory.max is 256 Mi, cpu.max corresponds to one core and restartCount stays at 0: the container did not restart. The Pod's status reflects the change: the applied limits and the allocated requests can be read in status.containerStatuses.

Terminal window
kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].allocatedResources}{"\n"}{.status.conditions[?(@.type=="PodResizeInProgress")]}{"\n"}'
{"limits":{"cpu":"1","memory":"256Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}
{"cpu":"100m","memory":"64Mi"}

The first line gives the applied resources, the second the allocated requests, and the third, empty, shows that no resize condition remains: the operation is complete. Now lower the memory limit to 96 Mi, above the current usage:

Terminal window
kubectl patch pod gourmand -n ressources --subresource resize --patch '{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"96Mi"}}}]}}' && sleep 5 && kubectl exec gourmand -n ressources -- cat /sys/fs/cgroup/memory.max
pod/gourmand patched
100663296

The decrease is applied: 100663296 is 96 Mi.

Going below the real usage is another case. The API accepts the patch, but the kubelet cannot apply a limit lower than what the container occupies. The Pod keeps its previous limit, no restart takes place, and the PodResizeInProgress condition stays True without a message. Going back to an applicable value clears the condition.

A request larger than what the node can host gives another state: the PodResizePending condition then carries a reason, such as Deferred when the node does not have enough free resources for the moment. On a 2-CPU node that is already busy, a request of 2 CPUs in requests and in limits gives, for example, this message in the condition:

Node didn't have enough resource: cpu, requested: 2000, used: 950, capacity: 2000

In this example, spec carries the new value (cpu 2) while status keeps the old ones (500m as a limit, 100m as a request), and restartCount stays at 0. In both cases, the Pod therefore keeps its old values.

Pending Pod: requests that nobody can serve

A Pod whose requests exceed what a node can offer is never placed. The trop-gros.yaml file asks for 64 CPUs and 256 Gi:

Terminal window
cat > trop-gros.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: trop-gros
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
resources:
requests:
cpu: "64"
memory: 256Gi
EOF
kubectl apply -f trop-gros.yaml -n ressources && sleep 8 && kubectl get pod trop-gros -n ressources
pod/trop-gros created
NAME READY STATUS RESTARTS AGE
trop-gros 0/1 Pending 0 8s

The Pod stays Pending: no container has started. The cause is in the scheduler's events:

Terminal window
kubectl get events -n ressources --field-selector involvedObject.name=trop-gros,reason=FailedScheduling -o custom-columns=MESSAGE:.message
MESSAGE
0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

The message says that the only node does not have enough unreserved CPU or memory. The cause is therefore read in the requests, not in the real consumption: a node can be almost empty and refuse a Pod that reserves too much. To compare the requests with the consumption, kubectl top pod needs metrics-server; on this cluster, it is not installed:

Terminal window
kubectl top pod -n ressources
error: Metrics API not available

Clean up the cluster

Deleting the namespace also deletes all the Pods it contains, including trop-gros, which stayed Pending:

Terminal window
kubectl delete namespace ressources
namespace "ressources" deleted

Troubleshooting

SymptomLikely causeWhat to do
OOMKilled, code 137The container exceeds its memory limitRaise the limit or reduce the usage
Pod Pending, Insufficient cpuThe requests exceed what the node offersLower the requests or add capacity
Slow application without a restartCPU throttling: the limit is reachedRaise or remove the CPU limit

Put it into practice

Two labs complete the demonstration. The first one has you declare resource budgets (requests and limits) for each of the two containers of the same Pod. The validation checks that the Pod runs with its two containers, that each budget is declared, and that the kernel enforces the memory limit of each container. The second one has you resize a Pod in place, without restarting it. The validation checks that the Pod asks for its new resources, that the resize happened without recreation or restart, and that the kubelet and the kernel apply the new limit.

Key takeaways

  1. A request is used for placement and guarantees a minimum; a limit is a ceiling.
  2. Above its CPU limit, a container is slowed down (throttling); above its memory limit, it is killed (OOMKilled, code 137).
  3. The QoS class is deduced from the resources: Guaranteed if requests equal limits, Burstable if some exist, BestEffort if none.
  4. Under memory pressure, the kubelet first evicts the Pods whose usage exceeds their requests, then takes the priority into account.
  5. The limit can be read from the container in /sys/fs/cgroup/memory.max and cpu.max with cgroup v2.
  6. kubectl patch pod --subresource resize changes the resources without restarting the container with resizePolicy: NotRequired; a decrease below the usage is not applied.
  7. A Pod Pending for Insufficient cpu or memory is read in the FailedScheduling events: the cause is in the requests.

Test your knowledge

This quiz covers what a CPU overrun and a memory overrun turn into, the deduction of the QoS class, and the in-place resize. Allow five minutes and aim for 80%; below that, reread "OOMKilled: when your Pod blows up".

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer