A denial of service in a Kubernetes cluster does not need a sophisticated attacker: a compromised application, or one that is simply badly written, with no limits can take all the CPU or all the memory of a node and starve the other applications. This lesson shows what a denial of service exhausts, how to read its symptoms, how to cap a namespace so that the load stays within its perimeter, and how to check that an application in another namespace keeps responding. It is for readers who already know requests, limits and quotas, and it prepares the "Denial of Service" objective of the "Kubernetes Threat Model" domain of the KCSA certification.
What you will learn
- Describe what a denial of service exhausts in a cluster.
- Cause and read CPU pressure, and recognise the symptoms of a memory load (evicted Pods, container killed for memory, node under pressure), which the demonstration does not cause.
- Cap a namespace with a ResourceQuota and a LimitRange, and check that a creation over quota is refused.
- Declare requests and limits on a container and say what happens when it exceeds its limit.
- Name the control plane components a denial of service can target.
- Check that an application in another namespace responds during the load.
- Detect a persistence left by an attacker and remove it without damaging the application, with the module challenge.
Prerequisites
- Have followed the lesson Persistence: what an attacker leaves behind (in French), because the end-of-module challenge combines the two.
- Know requests and limits: what a request guarantees, what a limit stops, and the QoS classes. This lesson does not explain them again.
- Know ResourceQuota and LimitRange: this lesson uses them on the defensive side and refers to their own lesson for the detail of each field.
- Know what a namespace is.
What is a denial of service in a cluster?
A denial of service exhausts a shared resource until other workloads can no longer use it. In the STRIDE method of the lesson Trust boundaries, it is the letter D. Here, the shared resource is the cluster itself. The table lists what can be exhausted, what you observe, and the control that opposes it.
| Targeted resource | What you observe | Control |
|---|---|---|
| CPU of a node | Pods that declare no requests slow down | requests.cpu and limits.cpu on every container, CPU quota of the namespace |
| Memory of a node | Container killed for memory, evicted Pods, node under pressure | limits.memory, memory quota of the namespace |
| Processes of a node (PID) | A fork bomb fills the process table, the node goes into the PIDPressure condition | Per-Pod process limit of the kubelet (podPidsLimit) |
| Local disk of a node | Logs or temporary files fill the disk, DiskPressure condition, evicted Pods | limits.ephemeral-storage quota of the namespace |
| Reservable capacity | A workload that declares very high requests exhausts what the scheduler can reserve, and new Pods stay Pending | requests.cpu and requests.memory quotas of the namespace |
| Number of Pods | The scheduler finds no room; a node accepts 110 Pods by default | pods quota of the namespace |
| Number of objects | etcd grows, the control plane slows down | count/... object quotas |
| Requests to the API server | kubectl commands and controllers answer slowly | API Priority and Fairness bounds the volume of requests, RBAC limits their cost |
| Storage | Saturated volumes, full etcd | requests.storage quota, etcd monitoring |
This table mixes two kinds of ground. CPU, memory, processes, disk and the number of Pods play out on the nodes and in the attacker's namespace: you handle them with caps. The number of objects and the requests to the API server hit the control plane, shared by the whole cluster, which a per-namespace cap protects only in part. Storage touches both. This lesson demonstrates the CPU pressure and the capping by quota; the other rows are read from the table, without any hands-on step.
What does a workload without limits cause?
A workload without limits takes what it can on its node, but not every neighbour suffers the same way. CPU is shared in proportion to requests: a neighbour that declares its requests keeps its share, even when the workload spins in a loop. A Pod without requests (BestEffort QoS class) gets a minimal weight, and it is the one that slows down.
The symptom then depends on the resource. A container that exceeds its memory limit is killed by the kernel. A container that exceeds its CPU limit is throttled. A container with no CPU limit can take the cores that the others do not reserve. When the node runs out of memory, it is the kubelet that evicts Pods, starting with those whose usage exceeds their requests: a greedy BestEffort Pod is therefore among the first candidates, with Pod priority breaking ties after that.
The attack has a second face: a workload that declares very high requests reserves the capacity of the node without necessarily consuming it. The scheduler then has no room left, and the new Pods of the other applications stay Pending. The number of Pods and objects created also weighs on the control plane (etcd, API server), even when the Pods do nothing.
To see these symptoms safely, start a load in a test namespace, never on a cluster that serves users. The following manifest deploys four Pods that run an endless loop, so each consumes one core, and declare no resources. In the example, the load lives in the essai-charge namespace and another application, the directory, in espace-annuaire: it is an ordinary web service, deployed with a Service of the same name, which the lab of this lesson already provides.
apiVersion: apps/v1kind: Deploymentmetadata: name: charge-gourmande namespace: essai-chargespec: replicas: 4 selector: matchLabels: app: charge-gourmande template: metadata: labels: app: charge-gourmande spec: containers: - name: boucle image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 command: ["sh", "-c", "while true; do :; done"]Create the namespace with kubectl create namespace essai-charge, apply the manifest, then measure the pressure. The commands below assume that metrics-server is installed: without it, kubectl top fails.
kubectl top nodeskubectl top pods -n essai-charge --sort-by=cpukubectl describe node | grep -A 8 'Conditions:'First look at the CPU% or MEMORY% column of kubectl top nodes: a node close to 100% is saturated. The second command then ranks the Pods by consumption and points at the culprit. This load only consumes CPU: you see CPU% rise, but no eviction. Neighbours that declare their requests keep their share of CPU, and those that declare none may slow down. The directory is therefore affected or not depending on its own requests.
The memory-related symptoms (OOMKilled, Evicted, MemoryPressure) are read with the same commands, but only on a workload that allocates memory. This lesson does not provide that manifest. In the output of describe node, the MemoryPressure line switches to True when the kubelet runs out of memory and starts evicting, and the evicted Pods then remain visible with the Failed status:
kubectl get pods -A --field-selector=status.phase=Failedkubectl get events -A --sort-by=.metadata.creationTimestampAn evicted Pod carries the reason Evicted in its status. A container killed for memory is read differently, in kubectl describe pod: the Last State section shows OOMKilled with exit code 137. These two symptoms have two different causes: OOMKilled hits the container that exceeded its own limit, while Evicted hits a Pod because the whole node is short of memory. The lesson Requests and limits causes an OOMKilled step by step.
How do you cap a namespace?
To cap a namespace, you set two objects together: a ResourceQuota that limits the total of the namespace, and a LimitRange that gives default values to containers that declare none. Both are needed. Without a LimitRange, a quota on CPU and memory refuses any Pod whose containers declare neither requests nor limits, and you block the legitimate applications that declared nothing. With one, they receive default values.
The following manifest caps essai-charge at 500 millicores and 512 MiB of memory requested, with 5 Pods at most, and provides default values. The fields are those of the lesson ResourceQuota and LimitRange, which you should consult for their detail.
apiVersion: v1kind: ResourceQuotametadata: name: plafond namespace: essai-chargespec: hard: pods: "5" requests.cpu: 500m requests.memory: 512Mi limits.cpu: "1" limits.memory: 1Gi---apiVersion: v1kind: LimitRangemetadata: name: valeurs-par-defaut namespace: essai-chargespec: limits: - type: Container defaultRequest: cpu: 100m memory: 64Mi default: cpu: 200m memory: 128MiWith these caps, the Pods created from now on cannot exceed one core of limit in total, even if the workload asks for more replicas. Apply the manifest with kubectl apply -f plafond.yaml, then read the state of the quota:
kubectl describe resourcequota plafond -n essai-chargeThe Used column is displayed next to Hard for each resource. It starts at zero if the namespace is empty, or counts what already runs.
Why is the workload already running not bounded?
A ResourceQuota and a LimitRange only apply to Pods created after them. They do not modify running Pods: a workload started before the cap keeps running without limits, and that is the trap of this lesson. To bound it, you must recreate its Pods: once deleted, they are restarted by their ReplicaSet, and the new ones then receive the default values.
kubectl delete pods -l app=charge-gourmande -n essai-chargekubectl get pods -n essai-charge \ -o custom-columns=NOM:.metadata.name,CPU_LIMIT:.spec.containers[0].resources.limits.cpuThe CPU_LIMIT column must show 200m for each new Pod. If it is empty, the Pod was created before the LimitRange.
For a few tens of seconds, the old Pods may stay in Terminating: the sh running as PID 1 does not listen for SIGTERM, and Kubernetes waits for the default grace period of 30 seconds before killing them. They still count in the quota, so you may see exceeded quota go by and fewer Pods than expected, until they disappear.
These default values are a safety net, not an answer: every manifest declares its own requests and limits, with values chosen for the application. For charge-gourmande, the following block goes under the boucle container. A container that exceeds its memory limit is then killed by the kernel, and a container that exceeds its CPU limit is throttled.
resources: requests: cpu: 100m memory: 64Mi limits: cpu: 200m memory: 128MiA Deployment that asks for more Pods than the quota allows is not refused at creation: it is its ReplicaSet that fails to create the extra Pods, and the exceeded quota message appears in the events of the namespace (kubectl get events -n essai-charge) and in kubectl describe replicaset. A Pod created directly is, on the other hand, refused immediately. The test Pod below asks for 2 cores, four times the requests.cpu cap:
kubectl apply -n essai-charge -f - <<'EOF'apiVersion: v1kind: Podmetadata: name: trop-grosspec: containers: - name: attente image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 command: ["sleep", "3600"] resources: requests: cpu: "2" memory: 64Mi limits: cpu: "2" memory: 128MiEOFThe server answers with a Forbidden error that contains exceeded quota: plafond, with the requested, used and limit values. This is the proof that the cap is active: you read it from a real creation attempt, not from reading the quota manifest.
Which control plane components can a denial of service target?
A denial of service can target the control plane rather than the nodes, and two components are exposed: the API server and etcd. The API server receives every request of the cluster, those of humans as well as those of controllers. A compromised account that floods it with requests, for example by listing all objects in a loop, can slow down the whole cluster. etcd stores every object: creating thousands of objects, even empty ones, makes it grow up to its storage quota (2 GiB by default), after which it accepts no more writes, as the lesson etcd details.
Three controls limit these attacks, each on a different point.
API Priority and Fairness bounds the volume of requests: the API server sorts them into queues and priority levels, so that a noisy source does not starve the others. It is a control plane setting, which the provider manages on a managed cluster.
RBAC limits their cost. A refused request is still received and processed, but an account without the right to list all Secrets or to create in bulk cannot trigger the heaviest requests. The lesson RBAC teaches it.
The object quota caps what a namespace can create. count/configmaps, count/secrets or count/pods in a ResourceQuota bound the number of objects, and therefore the size it can give to etcd.
How do you check that another application keeps responding?
To check isolation, call an application in another namespace while the load runs. Isolation is proven only by a real call: reading that the quota is set does not say that the directory responds. The following command starts a temporary Pod in the namespace of the directory and queries it by its service name.
kubectl get deployment annuaire -n espace-annuairekubectl run sonde -n espace-annuaire --rm -i --restart=Never \ --image=busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 \ -- wget -qO- -T 3 http://annuaireThe first command must show all replicas ready. The second must display the response of the application. If the 3-second timeout expires, the directory no longer responds: the load took too many resources on its node, or a NetworkPolicy blocks the probe. What the quota guarantees is that a workload whose Pods are created after it can no longer reserve or create beyond the cap: it does not exceed one core of limit in total, nor 5 Pods. If the directory slows down anyway, look at its own requests: without requests, it remains the first to suffer from a neighbour that consumes the node.
When you have finished, delete the test namespace with kubectl delete namespace essai-charge: the four Pods that spin in a loop go with it.
Which mistakes should you avoid?
Three mistakes come up again and again.
- Setting the quota without a LimitRange. Pods without requests or limits are then refused, and you block the legitimate applications instead of the abusive load.
- Believing that the quota protects Pods that are already running. It only applies to later creations.
- Capping the nodes and forgetting the control plane. An account that can create objects without limit threatens etcd, whatever the CPU cap.
Hands-on practice
The first lab is guided. A workload without limits runs in a test namespace. You cap it with a quota and default values, you prove that a creation over quota is refused and that a container that declares nothing is completed, you bound the running load, then you check that an application in another namespace keeps responding. The validation covers the state of the cluster, including the attempt to create over quota, read from a real try.
Challenge: detect and contain a persistence and a denial of service
The challenge is the unaided proof of the module, and it is graded on the state of the cluster you leave behind, not on the commands you typed. With no procedure, you find what an attacker left to keep their access, you record it before removing it for good without damaging the legitimate application, then you cap the namespace where an abusive load runs.
The validation reads the persistence first: the object is named in a findings file, it is deleted, the access does not come back even without the original Pod, and the legitimate application keeps all its replicas. It then reads the denial of service: the namespace carries a quota, a LimitRange gives default values, a creation over quota is refused on a real attempt, and an application in another namespace responds during the load.
While you work, dsoxlab check shows what is missing, without saying how to fix it, and the lesson to go back to is the one that teaches that state. The challenge builds on this lesson, on Persistence: what an attacker leaves behind (in French), on Requests and limits and on ResourceQuota and LimitRange.
Key takeaways
- A denial of service exhausts a shared resource: CPU and memory of the nodes, number of Pods, number of objects, requests to the API server, storage.
- A container that exceeds its memory limit is killed (
OOMKilled, exit code 137); a Pod is evicted (Evicted) when it is the node that runs out of memory. - You cap a namespace with two objects together: a ResourceQuota for the total, a LimitRange for the default values.
- These default values are a net: every manifest declares its own requests and limits.
- Neither the quota nor the LimitRange touches Pods that are already running: you must recreate them, for example by deleting the Pods so that their ReplicaSet restarts them.
- A creation over quota is refused with
exceeded quota; you prove it with a real attempt. - The API server and etcd are the exposed control plane components: API Priority and Fairness bounds the volume of requests, RBAC limits their cost, and object quotas cap what a namespace can create.
- A workload without limits mostly penalises neighbours without requests; a workload that reserves too much blocks the scheduler, and new Pods stay
Pending. - Isolation between namespaces is proven by calling the application of another namespace during the load, not by reading the quota.
Knowledge check
Check that you have grasped the essentials of this guide. The questions cover only what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: CIS Kubernetes Benchmark: audit your cluster security
A new full quiz with random questions
Work again only on the questions you missed