Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Upgrade a Kubernetes cluster without downtime

By updated

Improve this page

Logo Kubernetes

To upgrade a Kubernetes cluster, always follow the same order: control plane first, workers next, one node at a time. This guide details the upgrade strategy, the version skew policy that defines the version gaps allowed between components, and the complete workflow with kubeadm upgrade, from planning to the post-upgrade checks.

What you will learn

  • Understand the version skew policy: which version gaps are allowed
  • Follow the correct order for upgrading the components
  • Run a complete upgrade with kubeadm: plan, apply, drain, uncordon
  • Upgrade the worker nodes one by one without interruption
  • Check the cluster after the upgrade and handle failure cases
  • Prove the gesture without help with the module challenge: drain a node, put it back in service and upgrade the cluster

Prerequisites

Why upgrading is critical

Kubernetes publishes three minor versions a year, roughly one every four months. The project maintains the last three minor versions. Since 1.19, each version receives about one year of fixes, followed by two months of maintenance mode, about fourteen months in all (official patch releases calendar, read on 13 November 2026).

Staying on an obsolete version exposes your cluster to:

  • Unpatched vulnerabilities (CVEs)
  • Incompatibility with new versions of tools (Helm, Istio, operators)
  • Loss of technical support from cloud providers

The right strategy is to upgrade one minor version at a time, for example from 1.35 to 1.36, then from 1.36 to 1.37.

Understanding the version skew policy

The version skew policy defines the version gaps allowed between Kubernetes components. It is the fundamental rule to know before any upgrade.

Main rules

ComponentGap allowed relative to kube-apiserver
kube-apiserverreference component. In HA, the instances are upgraded one by one.
kubeletnever newer, up to three minor versions older
kube-proxynever newer, up to three minor versions older
kube-controller-manager / kube-schedulernever newer, up to one minor version older
kubectlone minor version of gap, newer or older

Two points of this table explain the upgrade strategy on their own. The first is the asymmetry: a kubelet can lag behind, never run ahead. The second is the size of the window, three versions for the nodes against only one for kubectl. This is what makes the two-phase upgrade possible, and a kubelet lagging behind its API server is a supported situation, not an anomaly to fix within the hour.

In an HA control plane, the kube-apiserver instances must be upgraded one by one, limiting the version gap between them. The other components must remain compatible with the oldest and the newest instance during the transition.

In practice

The version gap tolerance allows you to upgrade the control plane first, then the workers progressively, even on a cluster with many nodes. Always check the official version skew policy for the exact compatibility windows before you start.

Upgrade order with kubeadm

With kubeadm, the upgrade starts with the control plane through kubeadm upgrade apply, which handles the control plane components (API server, controller-manager, scheduler) as well as etcd and the addons when they are managed by kubeadm. If etcd is managed separately, its compatibility strategy must be checked separately.

Next, the workers are upgraded one by one with kubeadm upgrade node, followed by the kubelet upgrade.

Finally, upgrade kubectl on your administration workstations.

The diagram sums up this order. Spot the only node where apply is run, the first control plane: all the other nodes, control planes and workers alike, go through kubeadm upgrade node.

Upgrade order of a kubeadm cluster. First of all, a backup of etcd and of the control plane manifests. Step 1, the first control plane, with kubeadm upgrade apply run only once. Step 2, the other control planes, with kubeadm upgrade node. Step 3, the workers one by one, with kubeadm upgrade node. Step 4, the kubectl upgrade on the administration workstations.Before: back up etcdand the control plane manifests① First control planekubeadm upgrade apply, once only② Other control planeskubeadm upgrade node③ Workers, one at a timekubeadm upgrade node④ Admin workstationsupdate kubectl
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Before the upgrade: preparation

A completed version upgrade cannot be undone: the only way back is an etcd restore, with the loss of everything that changed since the backup. The three checks below are therefore not formalities, they are the only things you will have left if the operation goes wrong.

Check the cluster state

Make sure the cluster is healthy before you start:

Terminal window
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed

All nodes must be Ready. No critical pod must be in error.

Back up etcd

An etcd backup is essential before an upgrade. If something goes wrong, it lets you restore the cluster data from before the upgrade, on an etcd of the same major and minor version as the one that produced it.

Terminal window
sudo etcdctl snapshot save /tmp/etcd-backup-pre-upgrade.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key

Verify the snapshot with etcdutl and not etcdctl: the latter prints the help and exits with 0 without checking anything, which gives a backup reputed good without it ever having been read.

Terminal window
sudo etcdutl snapshot status /tmp/etcd-backup-pre-upgrade.db --write-out=table

The file contains all the cluster's Secrets: copy it off the node, and do not leave it in /tmp, which the system may empty on reboot. Also copy the control plane's /etc/kubernetes/manifests directory: without it, you could not go back to the manifests from before the upgrade.

These commands assume etcdctl and etcdutl are installed on the node. The lesson Back up and restore Kubernetes shows how to install them, then take and verify the snapshot.

Read the release notes

Before each upgrade, read the official Kubernetes release notes. Check:

  • The breaking changes and deprecated APIs
  • The new features reaching GA (stable)
  • The version-specific prerequisites

Read the release notes for the API removals, and not only for the new features. This is the point that breaks deployments without warning: a manifest that references a removed API version is rejected after the upgrade, while it worked the day before. The kubent or kubectl-convert tool lets you spot these references before switching.

Upgrading the control plane (kubeadm)

The upgrade is done on the first control plane node first, then on any other control plane nodes. The examples upgrade a cluster from 1.36.4 to 1.37.0, the upgrade that the lab at the end of the page plays.

Switch the package repository to the target version

The official pkgs.k8s.io repository publishes one repository per minor version: a node installed on 1.36 only sees the 1.36.x packages. Without this switch, apt-get install kubeadm=1.37.0-1.1 simply does not find the requested version. The Kubernetes documentation requires it before any minor upgrade, and the step is repeated on every node, workers included.

Terminal window
# The repository still points to the old minor version
cat /etc/apt/sources.list.d/kubernetes.list
# Make it point to the target minor version
sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list

The file must now contain /v1.37/:

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /

Step 1: upgrade kubeadm

On the control plane node, upgrade the kubeadm package to the target version:

Terminal window
# Ubuntu / Debian
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.37.0-1.1
sudo apt-mark hold kubeadm

Check that the installed version is indeed the target version:

Terminal window
kubeadm version

Step 2: plan the upgrade

This step changes nothing: it queries the cluster and prints the reachable versions and the components that will be touched. It is also the moment when kubeadm reports the incompatibilities it detects, before anything is committed.

Terminal window
sudo kubeadm upgrade plan

This command prints:

  • The current version of the cluster
  • The target version available
  • The components that will be upgraded
  • Any warnings (deprecated APIs, etc.)

Read the output carefully before going on.

Step 3: apply the upgrade

Terminal window
sudo kubeadm upgrade apply v1.37.0

The output first recalls the starting point and the target, then ends, if everything went well, with the SUCCESS line. Recorded on the lab cluster:

[upgrade/versions] Cluster version: v1.36.4
[upgrade/versions] kubeadm version: v1.37.0
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.0".

kubeadm upgrade apply upgrades the control plane components on the targeted node, and applies, depending on the cluster configuration, the upgrades of the components it manages: CoreDNS, kube-proxy and etcd.

Step 4: upgrade kubelet and kubectl

Terminal window
# Drain the node
kubectl drain k8s-cp.lab --ignore-daemonsets
# Upgrade the packages
sudo apt-mark unhold kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo apt-mark hold kubelet kubectl
# Restart the kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet
# Put the node back in service
kubectl uncordon k8s-cp.lab

Draining a control plane node depends on your architecture. In many clusters, no application workload runs there, thanks to the NoSchedule taint set on these nodes. Elsewhere, treat it like any node hosting Pods.

The two subcommands do not do the same job and are not used in the same place. kubeadm upgrade apply is only run once, on the first control plane node: it is the one that decides the target version and upgrades the control plane components. kubeadm upgrade node is then run on each of the other nodes, control planes and workers alike, and merely aligns the local configuration with the decision already made.

Upgrading the worker nodes

Workers are upgraded one by one to avoid any service interruption. For each worker, the workflow is identical, and it starts with the same repository switch to /v1.37/ as on the control plane.

  1. Upgrade kubeadm on the worker, repository switched:

    Terminal window
    sudo apt-mark unhold kubeadm
    sudo apt-get update && sudo apt-get install -y kubeadm=1.37.0-1.1
    sudo apt-mark hold kubeadm
  2. Apply the configuration:

    Terminal window
    sudo kubeadm upgrade node
  3. Drain the worker (from an administration workstation):

    Terminal window
    kubectl drain k8s-w1.lab --ignore-daemonsets --delete-emptydir-data
  4. Upgrade kubelet and kubectl:

    Terminal window
    sudo apt-mark unhold kubelet kubectl
    sudo apt-get update && sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
    sudo apt-mark hold kubelet kubectl
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet
  5. Put the worker back in service:

    Terminal window
    kubectl uncordon k8s-w1.lab
  6. Check before moving on to the next worker:

    Terminal window
    kubectl get nodes
    kubectl get pods -A -o wide | grep k8s-w1.lab

Post-upgrade checks

After upgrading all the nodes, check the whole cluster:

Node versions

The VERSION column of kubectl get nodes is the kubelet version of each node. To read only that, ask for it explicitly:

Terminal window
kubectl get nodes -o custom-columns=NOM:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion --no-headers

All nodes must show the target version. Recorded on the lab cluster, before then after the upgrade:

# before
k8s-cp.lab v1.36.4
k8s-w1.lab v1.36.4
# after
k8s-cp.lab v1.37.0
k8s-w1.lab v1.37.0

A node left on v1.36.4 almost always has a kubelet that was not replaced or not restarted: kubeadm upgrade never touches the kubelet package.

System pods

Terminal window
kubectl get pods -n kube-system

All the control plane pods must be Running.

Control plane health

Check the health of the components through the API server's readyz endpoint:

Terminal window
kubectl get --raw='/readyz?verbose' | grep '\[-\]\|\[\+\]'

Quick functional test

Create a test pod to validate that the cluster works:

Terminal window
kubectl run test-upgrade --image=nginx:1.30@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb --restart=Never
kubectl get pod test-upgrade -w

Wait for the pod to reach Running, then clean up:

Terminal window
kubectl delete pod test-upgrade

Handling an upgrade failure

The three situations below are ranked by increasing severity, and the response is not the same. A failing upgrade apply can be recovered: kubeadm rolls back, or you rerun the command. A kubelet that does not restart leaves a node out of service. An etcd restore loses everything that changed since the backup. Always try in this order.

kubeadm upgrade apply fails

When kubeadm upgrade apply fails along the way, kubeadm first attempts an automatic rollback. If it could not do so, for example after an abrupt shutdown of the machine, rerun the same command: it is idempotent, replaying it brings the real state back toward the state you ask for. The kubeadm documentation also offers sudo kubeadm upgrade apply --force to get out of a bad state, without changing the cluster version.

During the upgrade, kubeadm writes two backup folders under /etc/kubernetes/tmp:

  • kubeadm-backup-etcd-<date>-<time> contains a copy of the local etcd member's data, to put back in /var/lib/etcd;
  • kubeadm-backup-manifests-<date>-<time> contains the static Pod manifests, to put back in /etc/kubernetes/manifests.

These copies are only useful if the automatic rollback does not work. They stay on the node after the upgrade, and you must delete them by hand (source: Upgrading kubeadm clusters, section "Recovering from a failure state", read on 13 November 2026).

If the problem persists after the rerun, look at the kubelet log and the state of the node's containers:

Terminal window
sudo journalctl -u kubelet -n 50 --no-pager
sudo crictl ps -a

kubelet does not start after the upgrade

If the kubelet does not start on a node after the upgrade:

Terminal window
sudo journalctl -u kubelet -n 50 --no-pager

Frequent causes:

  • Version incompatibility: check that the installed versions respect the version skew policy and the upgrade order recommended by kubeadm
  • Outdated configuration: kubeadm upgrade node was not run before the kubelet upgrade
  • Certificates: the certificates have expired (renew with kubeadm certs renew all, then restart the four control plane static Pods, as described in the section Renewing certificates of the kubeadm guide)

Restore from the etcd backup

As a last resort, if the cluster is in an inconsistent state after a failed upgrade, restore the etcd backup. The complete procedure, with the order of the static Pods and the options to pass, is in the section How do you restore etcd into a new directory?. It ends with a step not to forget: updating etcd.local.dataDir in the kubeadm-config ConfigMap, so that the next upgrade does not bring back the old directory.

The snapshot can only be restored on an etcd of the same major and minor version as the one that produced it. If the upgrade changed the etcd version, also go back to the manifests from before the upgrade, otherwise the snapshot no longer matches the binary that reads it. This return to the manifests does not bring the kubelet back: if kubelet has already been upgraded, it becomes newer than the API server, which the Kubernetes version skew policy does not allow. This rollback therefore holds for a failure that occurred before the kubelet upgrade.

Troubleshooting

These symptoms are met during the operation, when the cluster is in an intermediate state, between two versions. Having read them in advance beats discovering them with a half-upgraded node and a maintenance window that is closing.

SymptomLikely causeSolution
apt-get install kubeadm=1.37.0-1.1 does not find the versionpkgs.k8s.io repository still on the previous minor versionSwitch kubernetes.list to /v1.37/, then apt-get update
Upgrade stuck for 10+ minutesSlow image pull or network timeoutCheck connectivity and pre-pull the images: kubeadm config images pull
Node stays NotReady after upgradekubelet not restarted or wrong versionsystemctl restart kubelet; check kubelet --version
Pods Pending after drainNot enough resources on the remaining nodesCheck requests/limits; uncordon first then drain again
CoreDNS no longer worksCustom CoreDNS ConfigMap overwrittenRestore the config from the backup; check kubectl -n kube-system get cm coredns
API server unreachable after upgradeExpired certificates or port in usesudo crictl logs <container-id> of the kube-apiserver container

Hands-on practice

This lab rebuilds the cluster: its installation brings it back to 1.36.4, otherwise there would be no upgrade to carry out. The upgrade to 1.37.0 is then done in order, the control plane first with its own command, the node afterwards, while an application keeps serving. The tests read the versions on each node and check that none was left drained of its Pods.

Challenge: drain a node, put it back in service and upgrade the cluster

The challenge is the unaided proof of the module "Maintain and update". It is graded on the state of the cluster you leave behind, not on the commands you typed. The kubeadm cluster runs one minor version behind, and an application serves with four replicas that must never fall below two available. You protect this application, you drain a worker then put it back in service, you keep a trace of the operation, you upgrade the whole cluster by one minor version, then you copy into the lab's git repository a change made directly on the cluster. No procedure is given to you.

The validation reads seven states, with no solution or value to copy. It checks the disruption budget, the draining of the worker and its return to service, the trace of the operation in a ConfigMap, the versions announced by the API server and then by each node, and the presence of the four replicas. Finally it compares the value in the cluster with the last commit of the repository. For the return to service, the grader starts an ephemeral Pod that the scheduler must place on the worker.

Passing requires 70 points out of 100, and there is no hint. Two states are blocking and must pass whatever the total: the drained worker, with the application Pods recreated elsewhere, and the whole cluster at 1.37.0, with no unschedulable node and all replicas ready. dsoxlab check lists the states that are not satisfied, without saying how to fix them: the lesson to go back to is the one that teaches that state. The challenge builds on this lesson, on Preparing a cluster maintenance (in French), on Managing nodes (in French) and on Maintenance and changes (in French), which puts the maintenance steps in order. Carrying a change made directly on the cluster back into the Git repository is taught in Modifying a resource in production (in French).

Key takeaways

  • Upgrade one minor version at a time: 1.36 → 1.37, never 1.35 → 1.37 directly.
  • With pkgs.k8s.io, switch the repository to the target minor version on each node before installing the packages.
  • The order is strict: control plane first (kubeadm upgrade apply), workers next (kubeadm upgrade node), one node at a time.
  • The version skew policy defines the compatibility windows between components. Check it systematically before an upgrade.
  • Back up etcd before each upgrade. It is your safety net.
  • Read the release notes to anticipate breaking changes and removed APIs.
  • kubeadm upgrade apply is idempotent: if kubeadm could not roll back by itself after a failure, rerun the same command.
  • Check the cluster after each node: kubectl get nodes, system pods, functional test.

Knowledge check

Seven questions on what cannot be recovered: the exact numbers of the version skew tolerance, the mandatory order that follows from it, and the fact that a finished upgrade cannot be undone.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer