A Pod killed with the OOMKilled message, another one stuck in Pending for lack of room, a third one that slows down without ever restarting: in all three cases, the cause is to be found in the requests and the limits. A request is the amount of CPU or memory that the scheduler reserves for the container in order to place it on a node. A limit is the ceiling that the container cannot exceed. This lesson teaches you to set these two values and read them back, to trigger then fix an OOMKilled, to resize a Pod without restarting it, and to understand why a Pod stays Pending.
To follow this lesson, you need to know Pods and have the demo cluster created in First cluster. The kubectl top command, which displays the real consumption, needs metrics-server, the component that collects these measurements. It is presented in Read the cluster state: kubectl get, describe, logs and top, and you do not need it for the demonstration.
What you will learn
- Write requests and limits and read them back with
kubectl describe - Read the QoS class of a Pod (
Guaranteed,BurstableorBestEffort) and understand what determines it - Recognize an
OOMKilled(code 137) and fix it by raising the memory limit - Read the applied limit from inside the container, in the cgroup
- Resize a Pod in place with
--subresource resize - Diagnose a
PendingPod caused by insufficient resources
Understand requests and limits
Both fields are set per container, in resources, for the CPU and the memory. The CPU is counted in millicores (100m is a tenth of a core) and the memory in bytes (64Mi).
Requests: the guaranteed minimum
The request is used for placement: the scheduler only places a Pod on a node that still has enough unreserved resources. It is also a guarantee: a container gets at least its request, even when the node is loaded.
Limits: the barrier not to cross
The limit is a ceiling enforced by the kernel, and it does not behave the same way for the two resources. Above its CPU limit, a container is slowed down: this is throttling. Above its memory limit, it is killed: this is the OOMKilled (Out Of Memory) status, with exit code 137.
Look at the three zones of the diagram: below the request, the resource is guaranteed; between the request and the limit, the container can expand if the node has capacity; above the limit, the CPU is slowed down and the memory is killed.
CPU throttling: when Kubernetes slows you down
A CPU limit of 500m grants 50,000 microseconds of CPU per 100,000-microsecond period: a container that consumes them before the end of the period waits for the next one. It is not killed, it is slowed down, and nothing shows up in the Pod's events. A slow Pod without any restart is the symptom to connect with a CPU limit that is too low, which the section on the cgroup lets you read back.
Read a Pod's resources
You work in a dedicated namespace, deleted at the end:
kubectl config use-context kind-demo && kubectl create namespace ressourcesSwitched to context "kind-demo".namespace/ressources createdThe names and messages of this lesson are in French on purpose, so that the outputs stay those of the tested cluster: ressources means "resources", garanti "guaranteed", extensible "expandable", libre "free", gourmand "greedy", trop-gros "too big", donnees "data", and the NOM column "NAME". The -corrige suffix of a file means "fixed". The echo messages "lecture de /dev/zero en mémoire" and "48 Mi en mémoire" mean "reading /dev/zero into memory" and "48 Mi in memory".
The qos.yaml file declares three nginx Pods: garanti has requests equal to the limits for the CPU and the memory, extensible has requests lower than a memory limit and no CPU limit, libre has no resources declared.
mkdir -p ~/atelier-k8s/ressources && cd ~/atelier-k8s/ressourcescat > qos.yaml <<'EOF'apiVersion: v1kind: Podmetadata: name: garantispec: containers: - name: app image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb resources: requests: cpu: 100m memory: 64Mi limits: cpu: 100m memory: 64Mi---apiVersion: v1kind: Podmetadata: name: extensiblespec: containers: - name: app image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb resources: requests: cpu: 50m memory: 32Mi limits: memory: 128Mi---apiVersion: v1kind: Podmetadata: name: librespec: containers: - name: app image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edbEOFkubectl apply -f qos.yaml -n ressources && kubectl wait --for=condition=Ready pod --all -n ressources --timeout=90spod/garanti createdpod/extensible createdpod/libre createdpod/extensible condition metpod/garanti condition metpod/libre condition metThe three Pods are ready. Kubernetes has put each one in a quality of service class (QoS), which it computes from the Pod's resources. Read it:
kubectl get pods -n ressources -o custom-columns=NOM:.metadata.name,QOS:.status.qosClassNOM QOSextensible Burstablegaranti Guaranteedlibre BestEffortThe three QoS classes
The three classes are explained by the resources of your three Pods:
Guaranteed: each container has, for the CPU and for the memory, a request and a limit that are equal and greater than zero. This is the case ofgaranti.Burstable: at least one request or one limit exists, without the Pod beingGuaranteed. This is the case ofextensible, which has no CPU limit and no request equal to its memory limit.BestEffort: no request and no limit at all. This is the case oflibre.
The class is not a setting: it is deduced from the resources that you declare. Now read the resources of the garanti Pod:
kubectl describe pod garanti -n ressources | grep -A8 "Limits:" Limits: cpu: 100m memory: 64Mi Requests: cpu: 100m memory: 64Mi Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ml6ct (ro)The Limits and Requests blocks are identical: this is what gives the Guaranteed class.
The class also tells how exposed a Pod is to eviction, but it is not the only criterion. When the node runs out of memory, the kubelet (the Kubernetes agent that runs on each node) evicts Pods, that is, it stops them to free room. It first picks the ones whose usage exceeds their requests, then it takes the Pod's priority into account. A BestEffort Pod, whose request is zero, always exceeds its request: it is therefore always among the eviction candidates.
OOMKilled: when your Pod blows up
The gourmand Pod reads /dev/zero into memory with tail, under a 32 Mi limit: it necessarily exceeds its limit.
cat > gourmand.yaml <<'EOF'apiVersion: v1kind: Podmetadata: name: gourmandspec: containers: - name: app image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 command: - sh - -c - | echo 'lecture de /dev/zero en mémoire' tail /dev/zero resources: requests: cpu: 100m memory: 32Mi limits: cpu: 500m memory: 32Mi # ❌ trop bas pour ce que le conteneur lit en mémoireEOFkubectl apply -f gourmand.yaml -n ressourcespod/gourmand createdThe comment in the manifest means "too low for what the container reads into memory". The container needs about twenty seconds to be killed, restarted and killed again. Wait 25 seconds before reading the state of the Pod, then the reason for the last termination of the container, which its status keeps:
sleep 25 && kubectl get pod gourmand -n ressources && kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'NAME READY STATUS RESTARTS AGEgourmand 0/1 OOMKilled 2 (20s ago) 25sOOMKilled 137The OOMKilled status and the restarts that pile up show that the container is killed, then restarted. The last line is OOMKilled 137: the kernel killed the container because it exceeded its memory limit, and 137 is 128 plus signal 9. kubectl describe pod gourmand -n ressources shows the same thing, with Reason: OOMKilled and Exit Code: 137 in the Last State section, then Restart Count: 2.
Fix the limit
kubectl apply does not change the limits of an existing Pod: here, delete the Pod and recreate it with a suitable limit (another way exists, see "Resize a Pod in place"). The gourmand-corrige.yaml file writes 48 MB into /dev/shm, a directory that lives in memory, under a 128 Mi limit. It also adds resizePolicy, which is used further on for the in-place resize.
cat > gourmand-corrige.yaml <<'EOF'apiVersion: v1kind: Podmetadata: name: gourmandspec: containers: - name: app image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 command: - sh - -c - | dd if=/dev/zero of=/dev/shm/donnees bs=1M count=48 echo "48 Mi en mémoire" sleep 3600 resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: NotRequired resources: requests: cpu: 100m memory: 64Mi limits: cpu: 500m memory: 128MiEOFkubectl delete pod gourmand -n ressources --wait=true && kubectl apply -f gourmand-corrige.yaml -n ressources && kubectl wait --for=condition=Ready pod/gourmand -n ressources --timeout=90s && sleep 5 && kubectl get pod gourmand -n ressourcespod "gourmand" deleted from ressources namespacepod/gourmand createdpod/gourmand condition metNAME READY STATUS RESTARTS AGEgourmand 1/1 Running 0 6sThe Pod is Running with 0 restarts: 48 MB fit under 128 Mi.
Read the limit from inside the container
The limit set in the manifest becomes a kernel setting, visible in the container's cgroup (the Linux control group that enforces the limit). On a node using cgroup v2, two files give it:
kubectl exec gourmand -n ressources -- sh -c 'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/cpu.max'13421772850000 100000134217728 is 128 Mi in bytes. 50000 100000 means 50,000 microseconds of CPU per 100,000-microsecond period: the 500m limit, that is half a core. Reading these files proves that the limit of the manifest is the one the kernel applies.
Resize a Pod in place
Until now, changing a resource meant recreating the Pod. Kubernetes 1.37 lets you resize a running container, through the resize subresource of the Pod. The resizePolicy field set above says that neither the CPU nor the memory requires a restart (NotRequired). The change goes through kubectl patch ... --subresource resize: the same patch without the subresource is refused, because a Pod only lets you change a few fields.
Raise the limits to 1 CPU and 256 Mi:
kubectl patch pod gourmand -n ressources --subresource resize --patch '{"spec":{"containers":[{"name":"app","resources":{"limits":{"cpu":"1","memory":"256Mi"}}}]}}'pod/gourmand patchedThe command is accepted. Wait for the kernel to apply the new limit, then read it back, with the number of restarts:
for i in $(seq 1 20); do [ "$(kubectl exec gourmand -n ressources -- cat /sys/fs/cgroup/memory.max)" = 268435456 ] && break; sleep 2; done; kubectl exec gourmand -n ressources -- sh -c 'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/cpu.max'; kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'268435456100000 1000000memory.max is 256 Mi, cpu.max corresponds to one core and restartCount stays at 0: the container did not restart. The Pod's status reflects the change: the applied limits and the allocated requests can be read in status.containerStatuses.
kubectl get pod gourmand -n ressources -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].allocatedResources}{"\n"}{.status.conditions[?(@.type=="PodResizeInProgress")]}{"\n"}'{"limits":{"cpu":"1","memory":"256Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}{"cpu":"100m","memory":"64Mi"}The first line gives the applied resources, the second the allocated requests, and the third, empty, shows that no resize condition remains: the operation is complete. Now lower the memory limit to 96 Mi, above the current usage:
kubectl patch pod gourmand -n ressources --subresource resize --patch '{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"96Mi"}}}]}}' && sleep 5 && kubectl exec gourmand -n ressources -- cat /sys/fs/cgroup/memory.maxpod/gourmand patched100663296The decrease is applied: 100663296 is 96 Mi.
Going below the real usage is another case. The API accepts the patch, but the kubelet cannot apply a limit lower than what the container occupies. The Pod keeps its previous limit, no restart takes place, and the PodResizeInProgress condition stays True without a message. Going back to an applicable value clears the condition.
A request larger than what the node can host gives another state: the PodResizePending condition then carries a reason, such as Deferred when the node does not have enough free resources for the moment. On a 2-CPU node that is already busy, a request of 2 CPUs in requests and in limits gives, for example, this message in the condition:
Node didn't have enough resource: cpu, requested: 2000, used: 950, capacity: 2000In this example, spec carries the new value (cpu 2) while status keeps the old ones (500m as a limit, 100m as a request), and restartCount stays at 0. In both cases, the Pod therefore keeps its old values.
Pending Pod: requests that nobody can serve
A Pod whose requests exceed what a node can offer is never placed. The trop-gros.yaml file asks for 64 CPUs and 256 Gi:
cat > trop-gros.yaml <<'EOF'apiVersion: v1kind: Podmetadata: name: trop-grosspec: containers: - name: app image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb resources: requests: cpu: "64" memory: 256GiEOFkubectl apply -f trop-gros.yaml -n ressources && sleep 8 && kubectl get pod trop-gros -n ressourcespod/trop-gros createdNAME READY STATUS RESTARTS AGEtrop-gros 0/1 Pending 0 8sThe Pod stays Pending: no container has started. The cause is in the scheduler's events:
kubectl get events -n ressources --field-selector involvedObject.name=trop-gros,reason=FailedScheduling -o custom-columns=MESSAGE:.messageMESSAGE0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.The message says that the only node does not have enough unreserved CPU or memory. The cause is therefore read in the requests, not in the real consumption: a node can be almost empty and refuse a Pod that reserves too much. To compare the requests with the consumption, kubectl top pod needs metrics-server; on this cluster, it is not installed:
kubectl top pod -n ressourceserror: Metrics API not availableClean up the cluster
Deleting the namespace also deletes all the Pods it contains, including trop-gros, which stayed Pending:
kubectl delete namespace ressourcesnamespace "ressources" deletedTroubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
OOMKilled, code 137 | The container exceeds its memory limit | Raise the limit or reduce the usage |
Pod Pending, Insufficient cpu | The requests exceed what the node offers | Lower the requests or add capacity |
| Slow application without a restart | CPU throttling: the limit is reached | Raise or remove the CPU limit |
Put it into practice
Two labs complete the demonstration. The first one has you declare resource budgets (requests and limits) for each of the two containers of the same Pod. The validation checks that the Pod runs with its two containers, that each budget is declared, and that the kernel enforces the memory limit of each container. The second one has you resize a Pod in place, without restarting it. The validation checks that the Pod asks for its new resources, that the resize happened without recreation or restart, and that the kubelet and the kernel apply the new limit.
Key takeaways
- A request is used for placement and guarantees a minimum; a limit is a ceiling.
- Above its CPU limit, a container is slowed down (throttling); above its memory limit, it is killed (
OOMKilled, code 137). - The QoS class is deduced from the resources:
Guaranteedif requests equal limits,Burstableif some exist,BestEffortif none. - Under memory pressure, the kubelet first evicts the Pods whose usage exceeds their requests, then takes the priority into account.
- The limit can be read from the container in
/sys/fs/cgroup/memory.maxandcpu.maxwith cgroup v2. kubectl patch pod --subresource resizechanges the resources without restarting the container withresizePolicy: NotRequired; a decrease below the usage is not applied.- A Pod
PendingforInsufficient cpuormemoryis read in theFailedSchedulingevents: the cause is in the requests.
Test your knowledge
This quiz covers what a CPU overrun and a memory overrun turn into, the deduction of the QoS class, and the in-place resize. Allow five minutes and aim for 80%; below that, reread "OOMKilled: when your Pod blows up".
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Debug a Kubernetes application
A new full quiz with random questions
Work again only on the questions you missed