Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Denial of service: exhausting a cluster

By updated

Improve this page

Logo Kubernetes

A denial of service in a Kubernetes cluster does not need a sophisticated attacker: a compromised application, or one that is simply badly written, with no limits can take all the CPU or all the memory of a node and starve the other applications. This lesson shows what a denial of service exhausts, how to read its symptoms, how to cap a namespace so that the load stays within its perimeter, and how to check that an application in another namespace keeps responding. It is for readers who already know requests, limits and quotas, and it prepares the "Denial of Service" objective of the "Kubernetes Threat Model" domain of the KCSA certification.

What you will learn

  • Describe what a denial of service exhausts in a cluster.
  • Cause and read CPU pressure, and recognise the symptoms of a memory load (evicted Pods, container killed for memory, node under pressure), which the demonstration does not cause.
  • Cap a namespace with a ResourceQuota and a LimitRange, and check that a creation over quota is refused.
  • Declare requests and limits on a container and say what happens when it exceeds its limit.
  • Name the control plane components a denial of service can target.
  • Check that an application in another namespace responds during the load.
  • Detect a persistence left by an attacker and remove it without damaging the application, with the module challenge.

Prerequisites

What is a denial of service in a cluster?

A denial of service exhausts a shared resource until other workloads can no longer use it. In the STRIDE method of the lesson Trust boundaries, it is the letter D. Here, the shared resource is the cluster itself. The table lists what can be exhausted, what you observe, and the control that opposes it.

Targeted resourceWhat you observeControl
CPU of a nodePods that declare no requests slow downrequests.cpu and limits.cpu on every container, CPU quota of the namespace
Memory of a nodeContainer killed for memory, evicted Pods, node under pressurelimits.memory, memory quota of the namespace
Processes of a node (PID)A fork bomb fills the process table, the node goes into the PIDPressure conditionPer-Pod process limit of the kubelet (podPidsLimit)
Local disk of a nodeLogs or temporary files fill the disk, DiskPressure condition, evicted Podslimits.ephemeral-storage quota of the namespace
Reservable capacityA workload that declares very high requests exhausts what the scheduler can reserve, and new Pods stay Pendingrequests.cpu and requests.memory quotas of the namespace
Number of PodsThe scheduler finds no room; a node accepts 110 Pods by defaultpods quota of the namespace
Number of objectsetcd grows, the control plane slows downcount/... object quotas
Requests to the API serverkubectl commands and controllers answer slowlyAPI Priority and Fairness bounds the volume of requests, RBAC limits their cost
StorageSaturated volumes, full etcdrequests.storage quota, etcd monitoring

This table mixes two kinds of ground. CPU, memory, processes, disk and the number of Pods play out on the nodes and in the attacker's namespace: you handle them with caps. The number of objects and the requests to the API server hit the control plane, shared by the whole cluster, which a per-namespace cap protects only in part. Storage touches both. This lesson demonstrates the CPU pressure and the capping by quota; the other rows are read from the table, without any hands-on step.

What does a workload without limits cause?

A workload without limits takes what it can on its node, but not every neighbour suffers the same way. CPU is shared in proportion to requests: a neighbour that declares its requests keeps its share, even when the workload spins in a loop. A Pod without requests (BestEffort QoS class) gets a minimal weight, and it is the one that slows down.

The symptom then depends on the resource. A container that exceeds its memory limit is killed by the kernel. A container that exceeds its CPU limit is throttled. A container with no CPU limit can take the cores that the others do not reserve. When the node runs out of memory, it is the kubelet that evicts Pods, starting with those whose usage exceeds their requests: a greedy BestEffort Pod is therefore among the first candidates, with Pod priority breaking ties after that.

The attack has a second face: a workload that declares very high requests reserves the capacity of the node without necessarily consuming it. The scheduler then has no room left, and the new Pods of the other applications stay Pending. The number of Pods and objects created also weighs on the control plane (etcd, API server), even when the Pods do nothing.

To see these symptoms safely, start a load in a test namespace, never on a cluster that serves users. The following manifest deploys four Pods that run an endless loop, so each consumes one core, and declare no resources. In the example, the load lives in the essai-charge namespace and another application, the directory, in espace-annuaire: it is an ordinary web service, deployed with a Service of the same name, which the lab of this lesson already provides.

apiVersion: apps/v1
kind: Deployment
metadata:
name: charge-gourmande
namespace: essai-charge
spec:
replicas: 4
selector:
matchLabels:
app: charge-gourmande
template:
metadata:
labels:
app: charge-gourmande
spec:
containers:
- name: boucle
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sh", "-c", "while true; do :; done"]

Create the namespace with kubectl create namespace essai-charge, apply the manifest, then measure the pressure. The commands below assume that metrics-server is installed: without it, kubectl top fails.

Terminal window
kubectl top nodes
kubectl top pods -n essai-charge --sort-by=cpu
kubectl describe node | grep -A 8 'Conditions:'

First look at the CPU% or MEMORY% column of kubectl top nodes: a node close to 100% is saturated. The second command then ranks the Pods by consumption and points at the culprit. This load only consumes CPU: you see CPU% rise, but no eviction. Neighbours that declare their requests keep their share of CPU, and those that declare none may slow down. The directory is therefore affected or not depending on its own requests.

The memory-related symptoms (OOMKilled, Evicted, MemoryPressure) are read with the same commands, but only on a workload that allocates memory. This lesson does not provide that manifest. In the output of describe node, the MemoryPressure line switches to True when the kubelet runs out of memory and starts evicting, and the evicted Pods then remain visible with the Failed status:

Terminal window
kubectl get pods -A --field-selector=status.phase=Failed
kubectl get events -A --sort-by=.metadata.creationTimestamp

An evicted Pod carries the reason Evicted in its status. A container killed for memory is read differently, in kubectl describe pod: the Last State section shows OOMKilled with exit code 137. These two symptoms have two different causes: OOMKilled hits the container that exceeded its own limit, while Evicted hits a Pod because the whole node is short of memory. The lesson Requests and limits causes an OOMKilled step by step.

How do you cap a namespace?

To cap a namespace, you set two objects together: a ResourceQuota that limits the total of the namespace, and a LimitRange that gives default values to containers that declare none. Both are needed. Without a LimitRange, a quota on CPU and memory refuses any Pod whose containers declare neither requests nor limits, and you block the legitimate applications that declared nothing. With one, they receive default values.

The following manifest caps essai-charge at 500 millicores and 512 MiB of memory requested, with 5 Pods at most, and provides default values. The fields are those of the lesson ResourceQuota and LimitRange, which you should consult for their detail.

apiVersion: v1
kind: ResourceQuota
metadata:
name: plafond
namespace: essai-charge
spec:
hard:
pods: "5"
requests.cpu: 500m
requests.memory: 512Mi
limits.cpu: "1"
limits.memory: 1Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: valeurs-par-defaut
namespace: essai-charge
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 64Mi
default:
cpu: 200m
memory: 128Mi

With these caps, the Pods created from now on cannot exceed one core of limit in total, even if the workload asks for more replicas. Apply the manifest with kubectl apply -f plafond.yaml, then read the state of the quota:

Terminal window
kubectl describe resourcequota plafond -n essai-charge

The Used column is displayed next to Hard for each resource. It starts at zero if the namespace is empty, or counts what already runs.

Why is the workload already running not bounded?

A ResourceQuota and a LimitRange only apply to Pods created after them. They do not modify running Pods: a workload started before the cap keeps running without limits, and that is the trap of this lesson. To bound it, you must recreate its Pods: once deleted, they are restarted by their ReplicaSet, and the new ones then receive the default values.

Terminal window
kubectl delete pods -l app=charge-gourmande -n essai-charge
kubectl get pods -n essai-charge \
-o custom-columns=NOM:.metadata.name,CPU_LIMIT:.spec.containers[0].resources.limits.cpu

The CPU_LIMIT column must show 200m for each new Pod. If it is empty, the Pod was created before the LimitRange.

For a few tens of seconds, the old Pods may stay in Terminating: the sh running as PID 1 does not listen for SIGTERM, and Kubernetes waits for the default grace period of 30 seconds before killing them. They still count in the quota, so you may see exceeded quota go by and fewer Pods than expected, until they disappear.

These default values are a safety net, not an answer: every manifest declares its own requests and limits, with values chosen for the application. For charge-gourmande, the following block goes under the boucle container. A container that exceeds its memory limit is then killed by the kernel, and a container that exceeds its CPU limit is throttled.

resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi

A Deployment that asks for more Pods than the quota allows is not refused at creation: it is its ReplicaSet that fails to create the extra Pods, and the exceeded quota message appears in the events of the namespace (kubectl get events -n essai-charge) and in kubectl describe replicaset. A Pod created directly is, on the other hand, refused immediately. The test Pod below asks for 2 cores, four times the requests.cpu cap:

Terminal window
kubectl apply -n essai-charge -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: trop-gros
spec:
containers:
- name: attente
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
resources:
requests:
cpu: "2"
memory: 64Mi
limits:
cpu: "2"
memory: 128Mi
EOF

The server answers with a Forbidden error that contains exceeded quota: plafond, with the requested, used and limit values. This is the proof that the cap is active: you read it from a real creation attempt, not from reading the quota manifest.

Which control plane components can a denial of service target?

A denial of service can target the control plane rather than the nodes, and two components are exposed: the API server and etcd. The API server receives every request of the cluster, those of humans as well as those of controllers. A compromised account that floods it with requests, for example by listing all objects in a loop, can slow down the whole cluster. etcd stores every object: creating thousands of objects, even empty ones, makes it grow up to its storage quota (2 GiB by default), after which it accepts no more writes, as the lesson etcd details.

Three controls limit these attacks, each on a different point.

API Priority and Fairness bounds the volume of requests: the API server sorts them into queues and priority levels, so that a noisy source does not starve the others. It is a control plane setting, which the provider manages on a managed cluster.

RBAC limits their cost. A refused request is still received and processed, but an account without the right to list all Secrets or to create in bulk cannot trigger the heaviest requests. The lesson RBAC teaches it.

The object quota caps what a namespace can create. count/configmaps, count/secrets or count/pods in a ResourceQuota bound the number of objects, and therefore the size it can give to etcd.

How do you check that another application keeps responding?

To check isolation, call an application in another namespace while the load runs. Isolation is proven only by a real call: reading that the quota is set does not say that the directory responds. The following command starts a temporary Pod in the namespace of the directory and queries it by its service name.

Terminal window
kubectl get deployment annuaire -n espace-annuaire
kubectl run sonde -n espace-annuaire --rm -i --restart=Never \
--image=busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 \
-- wget -qO- -T 3 http://annuaire

The first command must show all replicas ready. The second must display the response of the application. If the 3-second timeout expires, the directory no longer responds: the load took too many resources on its node, or a NetworkPolicy blocks the probe. What the quota guarantees is that a workload whose Pods are created after it can no longer reserve or create beyond the cap: it does not exceed one core of limit in total, nor 5 Pods. If the directory slows down anyway, look at its own requests: without requests, it remains the first to suffer from a neighbour that consumes the node.

When you have finished, delete the test namespace with kubectl delete namespace essai-charge: the four Pods that spin in a loop go with it.

Which mistakes should you avoid?

Three mistakes come up again and again.

  • Setting the quota without a LimitRange. Pods without requests or limits are then refused, and you block the legitimate applications instead of the abusive load.
  • Believing that the quota protects Pods that are already running. It only applies to later creations.
  • Capping the nodes and forgetting the control plane. An account that can create objects without limit threatens etcd, whatever the CPU cap.

Hands-on practice

The first lab is guided. A workload without limits runs in a test namespace. You cap it with a quota and default values, you prove that a creation over quota is refused and that a container that declares nothing is completed, you bound the running load, then you check that an application in another namespace keeps responding. The validation covers the state of the cluster, including the attempt to create over quota, read from a real try.

Challenge: detect and contain a persistence and a denial of service

The challenge is the unaided proof of the module, and it is graded on the state of the cluster you leave behind, not on the commands you typed. With no procedure, you find what an attacker left to keep their access, you record it before removing it for good without damaging the legitimate application, then you cap the namespace where an abusive load runs.

The validation reads the persistence first: the object is named in a findings file, it is deleted, the access does not come back even without the original Pod, and the legitimate application keeps all its replicas. It then reads the denial of service: the namespace carries a quota, a LimitRange gives default values, a creation over quota is refused on a real attempt, and an application in another namespace responds during the load.

While you work, dsoxlab check shows what is missing, without saying how to fix it, and the lesson to go back to is the one that teaches that state. The challenge builds on this lesson, on Persistence: what an attacker leaves behind (in French), on Requests and limits and on ResourceQuota and LimitRange.

Key takeaways

  • A denial of service exhausts a shared resource: CPU and memory of the nodes, number of Pods, number of objects, requests to the API server, storage.
  • A container that exceeds its memory limit is killed (OOMKilled, exit code 137); a Pod is evicted (Evicted) when it is the node that runs out of memory.
  • You cap a namespace with two objects together: a ResourceQuota for the total, a LimitRange for the default values.
  • These default values are a net: every manifest declares its own requests and limits.
  • Neither the quota nor the LimitRange touches Pods that are already running: you must recreate them, for example by deleting the Pods so that their ReplicaSet restarts them.
  • A creation over quota is refused with exceeded quota; you prove it with a real attempt.
  • The API server and etcd are the exposed control plane components: API Priority and Fairness bounds the volume of requests, RBAC limits their cost, and object quotas cap what a namespace can create.
  • A workload without limits mostly penalises neighbours without requests; a workload that reserves too much blocks the scheduler, and new Pods stay Pending.
  • Isolation between namespaces is proven by calling the application of another namespace during the load, not by reading the quota.

Knowledge check

Check that you have grasped the essentials of this guide. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer