Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

How Kubernetes worker nodes work

By updated

Improve this page

Logo Kubernetes

A worker Node is a machine of the cluster that runs your Pods. This lesson explains what runs on that machine: the kubelet, which starts the containers, the container runtime, which creates them, and kube-proxy, which makes Services reachable. You will also learn to read the state of a Node with kubectl, then tie together all the pieces of the module in a challenge. A failure exercise has you stop a worker with docker stop and wait about 6 minutes for the cluster to replace its Pods.

What you will learn

  • State the role of the kubelet, kube-proxy and the container runtime (CRI) on a Node.
  • Read the state of a Node (role, version, runtime, capacity, conditions) with kubectl get nodes -o wide and kubectl describe node.
  • Explain how the kubelet learns which Pods to launch and how it reports on them.
  • Describe what adding, draining and removing a Node do, without performing them.
  • Find out without help which component does what on a cluster, with the module challenge.

The prerequisites are the lesson The control plane and the four-node kind cluster demo from First cluster. The outputs in this lesson come from it.

Overview of a Worker Node

A worker Node carries three programs that do not overlap: one applies the decisions of the control plane, another creates the containers, the last one makes the Service network usable. None of the three decides anything: decisions come from the control plane.

A Worker Node: the kubelet reports the state to the API Server and goes through the CRI to make the container runtime (containerd, CRI-O) work and start the Pods; kube-proxy watches the EndpointSlices on the API Server to program the network.API Server(Control Plane)Worker Nodekubeletmain agent, manages Podskube-proxynetworking: Service rules, load balancingContainer Runtimecontainerd, CRI-O: containers, imagesCRI(Container Runtime Interface)PodsPod APod BPod C...reports statuswatch EndpointSlices
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

On the diagram, notice that the kubelet is the only component that talks to both the API Server and the runtime: it is the link between what the cluster wants and what runs on the machine.

ComponentRoleIt talks to
kubeletagent of the Node: starts and monitors the Pods assigned to itthe API Server, the runtime (CRI)
runtime (containerd)actually creates and runs the containersthe kubelet, through the CRI
kube-proxyprograms the network rules that make Services workthe API Server

The kubelet: the main agent

The kubelet turns the Pod descriptions received from the API Server into containers running on the machine. Without a kubelet, a Node is nothing more than an inert machine in the eyes of the cluster.

What the kubelet does

The kubelet repeats four missions in a loop. It watches the API Server to learn which Pods are assigned to its Node. It asks the runtime to create or delete the matching containers. It monitors the health of these containers (the probes, covered in the Probes guide). It reports the state of the Node and of the Pods to the API Server.

This loop explains a surprising behavior: a container killed by hand on the Node is restarted by the kubelet a few seconds later. On the other hand, a Pod deleted from a Deployment is not recreated by the kubelet: it is the ReplicaSet, on the control plane side, that creates a new one. The kubelet manages the containers of a Pod, the ReplicaSet manages the number of Pods.

Communication with the API Server

The kubelet follows the pull model, detailed in The control plane: it keeps a connection open to the API Server and receives a notification when a Pod is assigned to its Node (this is what is called a watch).

If the API Server becomes unreachable, the kubelet keeps running the Pods it has already received. It can no longer receive new ones, but applications already started keep answering.

The heartbeat: signaling the Node state

The kubelet regularly sends a sign of life, the heartbeat, to say that the Node is alive. In practice, every ten seconds it renews a Lease object named after the Node, in the kube-node-lease namespace. Here are the Lease objects of the cluster:

Terminal window
kubectl get lease -n kube-node-lease -o wide
Result
NAME HOLDER AGE
demo-control-plane demo-control-plane 3m49s
demo-worker demo-worker 3m36s
demo-worker2 demo-worker2 3m26s
demo-worker3 demo-worker3 3m26s

Each Node has its Lease, control plane included, and its holder (HOLDER) is the Node itself. Read the renewal date of one of them twice in a row, about ten seconds apart:

Terminal window
kubectl get lease -n kube-node-lease demo-worker -o jsonpath='{.spec.renewTime}'
Result
2026-11-10T22:25:48.829025Z
Terminal window
kubectl get lease -n kube-node-lease demo-worker -o jsonpath='{.spec.renewTime}'
Result
2026-11-10T22:25:58.846570Z

The date moved forward by ten seconds: the kubelet renewed its lease. When these renewals stop, the Node is no longer reachable. On kind, you can cause this by stopping the container of a Node (nothing is destroyed), which takes about six minutes of waiting in all. The exercise only shows something if the stopped Node carries a Pod of the web Deployment (recreate it with the command from the lesson Kubernetes architecture if it no longer exists). First read the NODE column:

Terminal window
kubectl get pods -l app=web -o wide

Then stop a worker that carries a web Pod: demo-worker3 is the one in these outputs, replace it with yours if needed.

Terminal window
docker stop demo-worker3
kubectl get nodes
Result
demo-worker3
NAME STATUS ROLES AGE VERSION
demo-control-plane Ready control-plane 4m v1.37.0
demo-worker Ready <none> 3m46s v1.37.0
demo-worker2 Ready <none> 3m46s v1.37.0
demo-worker3 Ready <none> 3m46s v1.37.0

The Node is still Ready right after the stop: the cluster does not know yet. To follow the evolution, rerun kubectl get nodes every ten seconds, or let kubectl get nodes -w run (Ctrl+C stops it). On this cluster, sampled every ten seconds, Ready was still True 41 seconds after the stop and Unknown 51 seconds after, which matches the default delay of about 50 seconds (node-monitor-grace-period). About a minute after the stop, kubectl get nodes shows:

Terminal window
kubectl get nodes
Result
NAME STATUS ROLES AGE VERSION
demo-control-plane Ready control-plane 4m51s v1.37.0
demo-worker Ready <none> 4m37s v1.37.0
demo-worker2 Ready <none> 4m37s v1.37.0
demo-worker3 NotReady <none> 4m37s v1.37.0

The Node demo-worker3 is NotReady. In kubectl describe node demo-worker3, its four conditions show Unknown with the message Kubelet stopped posting node status., and the Node receives a node.kubernetes.io/unreachable taint (a mark set on a Node that tells the cluster to stop sending Pods there and to move away those that have no toleration for it). Here is the relevant excerpt, taken about a minute after the stop:

Terminal window
kubectl describe node demo-worker3
Result (excerpt)
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
[...]
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Tue, 10 Nov 2026 22:51:24 +0000 Tue, 10 Nov 2026 22:52:23 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 10 Nov 2026 22:51:24 +0000 Tue, 10 Nov 2026 22:52:23 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 10 Nov 2026 22:51:24 +0000 Tue, 10 Nov 2026 22:52:23 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 10 Nov 2026 22:51:24 +0000 Tue, 10 Nov 2026 22:52:23 +0000 NodeStatusUnknown Kubelet stopped posting node status.

The two taints and the four conditions at Unknown confirm that the cluster no longer hears from the kubelet.

The Pods of this Node are only replaced after an additional delay: each Pod carries by default a toleration of 300 seconds for the unreachable taint (it accepts to stay on the Node for that delay), visible in its kubectl describe pod (node.kubernetes.io/unreachable:NoExecute op=Exists for 300s), that is five minutes. This slowness is deliberate: a briefly unstable network must not trigger the move of every application on a machine.

On this cluster, the replacement appeared about five minutes after the switch to Unknown, a little under six minutes after the stop. Read the Pods of the web Deployment at that moment:

Terminal window
kubectl get pods -l app=web -o wide
Result
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-6d8c9555f7-8czhv 1/1 Terminating 0 6m27s 10.244.1.6 demo-worker3 <none> <none>
web-6d8c9555f7-fz6qm 1/1 Running 0 2s 10.244.2.4 demo-worker <none> <none>
web-6d8c9555f7-v2l4n 1/1 Running 0 6m27s 10.244.3.3 demo-worker2 <none> <none>
web-6d8c9555f7-x8j6s 1/1 Running 0 6m27s 10.244.2.3 demo-worker <none> <none>

Look at the first line: the old Pod is Terminating on demo-worker3, and its replacement web-6d8c9555f7-fz6qm, created two seconds earlier, is already running on demo-worker. The old Pod stays Terminating until the Node comes back: the kubelet, the only one able to stop its container, is unreachable, so the cluster cannot confirm its deletion. This often surprises people, because kubectl displays both the old Pod and its replacement. Restart the Node:

Terminal window
docker start demo-worker3
Result
demo-worker3

Docker answers with the name of the started container. Wait about twenty seconds, until kubectl get nodes shows Ready again for demo-worker3, then read the Pods again:

Terminal window
kubectl get pods -l app=web -o wide
Result
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-6d8c9555f7-fz6qm 1/1 Running 0 2m24s 10.244.2.4 demo-worker <none> <none>
web-6d8c9555f7-v2l4n 1/1 Running 0 8m49s 10.244.3.3 demo-worker2 <none> <none>
web-6d8c9555f7-x8j6s 1/1 Running 0 8m49s 10.244.2.3 demo-worker <none> <none>

Once the Node is Ready again, the old Pod is gone: the three requested Pods remain. The replacement Pod did not go back to demo-worker3, it stayed on demo-worker, because the ReplicaSet does not move a Pod that is already running.

The CRI: Container Runtime Interface

The kubelet never creates a container itself. It sends a standardized request to a container runtime, a program whose job this is (containerd, for example). The interface that standardizes this request is called the CRI (Container Runtime Interface), and it is what allows replacing one runtime with another without touching the kubelet.

Container execution flow

When a Pod is assigned to the Node, the kubelet calls the runtime through the CRI, over gRPC (a protocol for calling functions between programs). The runtime downloads the image if it is absent, then creates and starts the container.

Container creation chain: the kubelet watches the desired state on the API Server, calls the CRI over gRPC, containerd takes over through a shim then has runc run the container.1. watch2. gRPC3. shim4. execAPI Serverdesired state (Pods)kubeletorchestrates locallyCRIstandard interfacecontainerdhigh-level runtimerunclow-level runtimeContainer
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Read the diagram from left to right: the API Server notifies the kubelet, the kubelet calls the CRI, containerd takes over and hands the creation of the container to runc, the low-level program that relies on the isolation features of the Linux kernel. You do not have to call runc yourself: remember that the kubelet delegates everything to the runtime.

CRI-compatible runtimes

Kubernetes ships no runtime. containerd is the one kind embeds, and the one you generally install before kubeadm, which installs none. CRI-O is a second runtime designed for Kubernetes, used notably by OpenShift.

The removal of dockershim in Kubernetes 1.24 led people to wrongly say that Docker no longer worked. What Kubernetes removed is the internal bridge that let the kubelet drive Docker Engine without going through the CRI. Your Docker images still work, since they follow the standard OCI format, and Docker Engine remains usable with the cri-dockerd adapter. For a new cluster, prefer containerd: it is what Docker itself uses underneath.

kube-proxy: the network manager

kube-proxy programs on every Node the network rules that make Services work. A Service is a stable address placed in front of a group of Pods, presented in Services.

What kube-proxy does

Despite its name, kube-proxy does not see the traffic go by. It watches Services and their EndpointSlices (the list of ready Pods behind each Service) through the API Server, then programs the Linux kernel so that it redirects packets itself to one of those Pods. A stopped kube-proxy therefore does not cut existing connections: the rules already installed keep applying, but they no longer follow changes.

It does not make Pods communicate with each other either: the network plugin (CNI) does that. Without kube-proxy, Services would stop answering, and Pods would keep reaching each other by their IP addresses.

Which kube-proxy mode is used by default?

The mode is read in the kube-proxy configuration, stored in a ConfigMap:

Terminal window
kubectl -n kube-system get configmap kube-proxy -o yaml | grep 'mode:'
Result
mode: iptables

kube-proxy therefore programs iptables rules, the default mode on Linux, and this is the case on your cluster. The other modes belong to Pod networking.

kube-proxy as a DaemonSet

kube-proxy runs as a DaemonSet, an object that asks for one Pod on every Node. Kubernetes therefore places one on the control plane too, thanks to a toleration this DaemonSet carries, and one more on every Node that joins the cluster. Check it on demo:

Terminal window
kubectl get pods -n kube-system -o wide
Result (excerpt)
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kindnet-cfxwq 1/1 Running 1 (22s ago) 11m 172.18.0.2 demo-worker3 <none> <none>
kindnet-clf52 1/1 Running 0 11m 172.18.0.3 demo-worker <none> <none>
kindnet-q92qc 1/1 Running 0 12m 172.18.0.4 demo-control-plane <none> <none>
kindnet-r4tmw 1/1 Running 0 11m 172.18.0.5 demo-worker2 <none> <none>
[...]
kube-proxy-2ctdx 1/1 Running 0 11m 172.18.0.3 demo-worker <none> <none>
kube-proxy-52hw6 1/1 Running 0 11m 172.18.0.5 demo-worker2 <none> <none>
kube-proxy-d7sn9 1/1 Running 0 12m 172.18.0.4 demo-control-plane <none> <none>
kube-proxy-qzdvk 1/1 Running 1 (22s ago) 11m 172.18.0.2 demo-worker3 <none> <none>

Look at the NODE column: there is exactly one kube-proxy Pod per Node, control plane included, and the same goes for kindnet, kind's network plugin. The omitted lines ([...]) are CoreDNS and the control plane components. The two Pods of demo-worker3 (kindnet-cfxwq and kube-proxy-qzdvk) show RESTARTS 1: they restarted with that Node during the failure exercise in the heartbeat section.

How do you read the state of a Node with kubectl?

A Node is read with two commands: kubectl get nodes -o wide for the overview of all the Nodes, kubectl describe node for the detail of a single one. They answer different questions, and that is what the two outputs below show.

Terminal window
kubectl get nodes -o wide
Result
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
demo-control-plane Ready control-plane 12m v1.37.0 172.18.0.4 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-worker Ready <none> 11m v1.37.0 172.18.0.3 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-worker2 Ready <none> 11m v1.37.0 172.18.0.5 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-worker3 Ready <none> 11m v1.37.0 172.18.0.2 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4

Three columns answer the questions of this lesson. STATUS says whether the kubelet reports a ready Node (Ready). VERSION gives the kubelet version, here v1.37.0. CONTAINER-RUNTIME names the runtime and its version: containerd://2.3.4 on all four Nodes. The ROLES column shows <none> for workers, which is normal: Kubernetes puts no role label on a worker Node. KERNEL-VERSION is that of the host machine, since kind Nodes share its kernel.

For capacity, query the relevant fields directly. The -o custom-columns option displays only the chosen fields, one per column, in the form TITLE:path.to.field. This is what the Scheduler consults to know whether a Pod fits on a Node:

Terminal window
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memory,PODS:.status.capacity.pods
Result
NAME CPU MEM PODS
demo-control-plane 2 4009776Ki 110
demo-worker 2 4009776Ki 110
demo-worker2 2 4009776Ki 110
demo-worker3 2 4009776Ki 110

The four Nodes show the same capacity, 2 CPUs and 4009776Ki of memory, because they are containers that share the resources of the host machine. On a real cluster, each Node has its own. The PODS column is 110 by default: it is the maximum number of Pods per Node.

For the detail of a Node, kubectl describe node displays everything the cluster knows about it. The output is long: here are the parts to read, the [...] mark what is omitted.

Terminal window
kubectl describe node demo-worker2
Result (excerpt)
Name: demo-worker2
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=demo-worker2
kubernetes.io/os=linux
topology.kubernetes.io/zone=zone-b
[...]
Taints: <none>
Unschedulable: false
[...]
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 10 Nov 2026 22:31:24 +0000 Tue, 10 Nov 2026 22:22:14 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 10 Nov 2026 22:31:24 +0000 Tue, 10 Nov 2026 22:22:14 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 10 Nov 2026 22:31:24 +0000 Tue, 10 Nov 2026 22:22:14 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 10 Nov 2026 22:31:24 +0000 Tue, 10 Nov 2026 22:22:29 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.18.0.5
Hostname: demo-worker2
Capacity:
cpu: 2
ephemeral-storage: 31491104768
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4009776Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 31491104768
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4009776Ki
pods: 110
System Info:
[...]
Kernel Version: 6.12.115+deb13.1-amd64
OS Image: Debian GNU/Linux 13 (trixie)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://2.3.4
Kubelet Version: v1.37.0
[...]

Read the Capacity block and the Allocatable block together: Capacity is what the machine has, Allocatable is what the Scheduler can actually hand out to Pods, once the share reserved for the system is removed. Here the two are identical, because the kind Node reserves nothing. The System Info block repeats the runtime (Container Runtime Version) and the kubelet version (Kubelet Version) seen above. Taints: <none> says that this worker accepts any Pod, unlike the control plane.

For the conditions, the first three signal a problem when their status is True: lack of memory, disk or processes. The last one, Ready, does the opposite, and True is what you expect. A healthy Node therefore shows False, False, False, True. As soon as a status departs from this pattern, the Message column says why, and this is the starting point of a diagnosis, covered in node troubleshooting.

Add, drain or remove a node

A cluster evolves: you add a Node when the load grows, you take one out of service to update it, you remove one for good. This lesson does not have you perform them, but here is what each gesture does, because they all rely on the split of roles you have just seen.

GestureWhat it doesWhere it is covered
kubeadm joina new Node registers with the API Server, its kubelet starts and Pods can be assigned to itJoin a worker to a kubeadm cluster
kubectl cordonthe Node no longer accepts new Pods, those already running stayManage nodes (in French)
kubectl drainstarts with a cordon, then evicts the Pods of the Node: their ReplicaSets replace them with new Pods on the other NodesManage nodes (in French)
kubectl delete nodethe Node object disappears from the API: the cluster no longer counts itManage nodes (in French)

The common point is reconciliation: after a drain, the Pods managed by a ReplicaSet are not lost, because the ReplicaSet notices that replicas are missing and replaces them with new Pods on the remaining Nodes, with a new decision by the Scheduler (they are not the same Pods moved). A Node failure follows the same path, after the five-minute delay seen above. A kubelet that no longer starts belongs to node troubleshooting, not to this lesson.

Put it into practice

Two exercises, from guided to unassisted. The guided lab has you read a worker Node: you record its runtime, the version of its kubelet, its capacity and the state of its Ready condition, with the commands of the previous section. The lab designates the worker to read and has you write its sheet in ~/lire-un-node/fiche.txt. The validation compares your findings with what the cluster says.

The second is the unassisted proof of the module. The quiz checks that you recognize the right answer, the challenge checks that you know how to do it: it gives an objective, without the commands, and you find out on your own, on a kind cluster, which component carries which function. You record your findings in a file ~/architecture/releve.txt, in key: value format. It is graded on the state of the cluster you leave behind, not on the commands typed.

The challenge asks you to find and record:

  • the Node that carries the control plane;
  • the four Pods of the control plane components, with their role;
  • the runtime of each worker Node;
  • the Node that received the temoin Pod, with the Scheduled event that proves it;
  • the state of an already present Deployment whose Pods stay Pending without a Scheduled event: who created these Pods, and which component did not act.

To name a component, read the From column of the Events section of kubectl describe: it gives its exact name. Each piece of information corresponds to a lesson of the module: the architecture, the control plane and this lesson.

While you work, dsoxlab check lists what still fails. A failing state tells you what is missing, without saying how to fix it. The lesson to go back to is the one that teaches that state: reread it, redo the gesture on your cluster, then rerun dsoxlab check.

When you have finished the module, challenge included, delete the demonstration Deployment, then the demo cluster to free the memory:

Terminal window
kubectl delete deployment web
kind delete cluster --name demo
Result
deployment.apps "web" deleted from default namespace
Deleting cluster "demo" ...
Deleted nodes: ["demo-worker2" "demo-worker" "demo-worker3" "demo-control-plane"]

The second command confirms the deletion of the four Nodes. The cluster is recreated by following First cluster when you need it again.

Key takeaways

  • A worker Node carries three programs: the kubelet, a container runtime (containerd) and kube-proxy, and none of them makes decisions.
  • The kubelet learns its Pods by watching the API Server (pull model) and their state goes back the same way.
  • The kubelet restarts a container that dies, but it is the ReplicaSet that recreates a deleted Pod.
  • The kubelet does not create containers: it goes through the CRI, which lets you change runtime without touching it.
  • The removal of dockershim in 1.24 did not stop your Docker images from running: the image format is the same.
  • kube-proxy programs the kernel for Services (iptables by default) and runs as a DaemonSet, one Pod per Node.
  • kubectl get nodes -o wide gives the status, version and runtime; kubectl describe node gives the conditions and capacity.
  • A Node that stops sending a heartbeat goes NotReady after about fifty seconds, and its Pods are replaced about five minutes later.

Knowledge check

The questions cover what each component of a Node does, the direction of communications with the API Server and how to read the state of a Node.

Knowledge check

Check what you have learned with this interactive quiz

10 questions
5 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer