Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes rolling updates and rollbacks: tune and undo

By updated

Improve this page

Logo Kubernetes

A rolling update replaces the Pods of a Deployment in small groups: the new ones start while the old ones still serve. This lesson shows how to tune the speed of this replacement (maxSurge, maxUnavailable), how to switch from the Recreate strategy to RollingUpdate, how to detect a stuck rollout with progressDeadlineSeconds, and how to read, annotate and undo the revision history. You play it all on a Deployment of five replicas.

Prerequisites: the lesson Deployments, which creates a Deployment, ships a first update and undoes a wrong image, and the demo cluster from First cluster. This lesson does not repeat those actions: it tunes and goes deeper.

What you will learn

  • Trigger an update with kubectl set image or by changing the manifest, and follow it with kubectl rollout status
  • Tune maxSurge and maxUnavailable, and predict the number of Pods during the switch
  • Switch from Recreate to RollingUpdate and read the difference in the events
  • Detect a stuck rollout with progressDeadlineSeconds and go back with kubectl rollout undo
  • Read and annotate the revision history (change-cause)
  • Group several changes into a single rollout with the pause

How does a rolling update work?

At every change to the Pod template, the Deployment creates a new ReplicaSet. It then increases the number of Pods of the new ReplicaSet and decreases that of the old one, in steps, as long as two guardrails are respected: maxSurge (how many Pods above the desired number are tolerated) and maxUnavailable (how many unavailable Pods are tolerated). Both are 25% by default.

Rolling update of a Deployment of three Pods: the ReplicaSet of version 1 goes from three Pods to zero while the one of version 2 goes from zero to three, keeping three Pods available at each step.Initial stateStep 1Step 2Final stateRS v1: 3 PodsRS v1: 2 PodsRS v2: 1 Pod= 3 Pods availableRS v1: 1 PodRS v2: 2 Pods= 3 Pods availableRS v1: 0 PodRS v2: 3 Podsv1 kept for rollback
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

The diagram shows the example of three Pods: at each step a Pod of version 2 appears before a Pod of version 1 disappears, and the old ReplicaSet stays at zero at the end, ready for a rollback. Only a change to the Pod template (image, environment variable, template label) triggers this mechanism: changing replicas creates no ReplicaSet.

The calculation is simple. With 5 replicas, maxSurge: 2 and maxUnavailable: 1, there will be at any moment at most 5 + 2 = 7 Pods, and at least 5 - 1 = 4 Pods available. A value can be written as a number or as a percentage of the number of replicas.

How does the Recreate strategy behave?

The Recreate strategy deletes all the old Pods before creating the new ones: the application is cut for a while, but two versions never run together. Create the shop Deployment of five replicas with this strategy, then update it:

Terminal window
kubectl config use-context kind-demo
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
spec:
replicas: 5
strategy:
type: Recreate
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
EOF
kubectl rollout status deployment/shop --timeout=90s
kubectl set image deployment/shop nginx=nginx:1.30.6@sha256:42c68eab6f7e22611786a9fe1d1c019e275c50185bd54b8f75dc2b9184322703
kubectl rollout status deployment/shop --timeout=90s
kubectl describe deployment shop | sed -n '/^Events:/,$p'
Output
Switched to context "kind-demo".
deployment.apps/shop created
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
deployment.apps/shop image updated
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 0 of 5 updated replicas are available...
[...]
deployment "shop" successfully rolled out
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 36s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 5
Normal ScalingReplicaSet 20s deployment-controller Scaled down replica set shop-56f6b69df8 from 5 to 0
Normal ScalingReplicaSet 19s deployment-controller Scaled up replica set shop-685d5d7bc5 from 0 to 5

Read the order of the events: the old ReplicaSet goes down from 5 to 0, and only then does the new one go up from 0 to 5. During the interval, no Pod serves: this is the price of the Recreate strategy, which suits cases where two versions cannot coexist (for example because of a database schema).

How do you switch to RollingUpdate with maxSurge and maxUnavailable?

Now switch to RollingUpdate, with maxSurge: 2 and maxUnavailable: 1, by changing the manifest. So that this change also ships a new version, the manifest goes back to the starting image:

Terminal window
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
spec:
replicas: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
EOF
kubectl rollout status deployment/shop --timeout=90s
kubectl get deployment shop -o jsonpath='{.spec.strategy}{"\n"}'
kubectl get replicaset -l app=shop
kubectl describe deployment shop | sed -n '/^Events:/,$p'
Output
deployment.apps/shop configured
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 2 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 1 old replicas are pending termination...
[...]
deployment "shop" successfully rolled out
{"rollingUpdate":{"maxSurge":2,"maxUnavailable":1},"type":"RollingUpdate"}
NAME DESIRED CURRENT READY AGE
shop-56f6b69df8 5 5 5 38s
shop-685d5d7bc5 0 0 0 21s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 39s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 5
Normal ScalingReplicaSet 23s deployment-controller Scaled down replica set shop-56f6b69df8 from 5 to 0
Normal ScalingReplicaSet 22s deployment-controller Scaled up replica set shop-685d5d7bc5 from 0 to 5
Normal ScalingReplicaSet 3s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 2
Normal ScalingReplicaSet 3s deployment-controller Scaled down replica set shop-685d5d7bc5 from 5 to 4
Normal ScalingReplicaSet 2s deployment-controller Scaled up replica set shop-56f6b69df8 from 2 to 3
Normal ScalingReplicaSet 2s deployment-controller Scaled down replica set shop-685d5d7bc5 from 4 to 3
Normal ScalingReplicaSet 2s deployment-controller Scaled up replica set shop-56f6b69df8 from 3 to 4
Normal ScalingReplicaSet 1s deployment-controller Scaled down replica set shop-685d5d7bc5 from 3 to 2
Normal ScalingReplicaSet 1s (x3 over 1s) deployment-controller (combined from similar events): Scaled down replica set shop-685d5d7bc5 from 1 to 0

The first three events are those of the previous step. The following ones alternate between the two ReplicaSets: the new one goes from 0 to 2 while the old one goes from 5 to 4, then from 2 to 3 while the old one goes to 3, and so on down to zero. Between two steps, the Deployment stays within its bounds: 7 Pods at most, 4 available at least.

Also notice that there are only two ReplicaSets at the end, although the image took three successive values (1.30.5, 1.30.6, then 1.30.5 again). The kubectl apply reactivated the ReplicaSet of image 1.30.5, created at the start: a ReplicaSet whose Pod template is identical is reused, not recreated. The settings read as follows:

SettingValue in the exampleEffect
maxSurge2Up to 7 Pods for 5 desired: the replacement goes faster, at the cost of extra resources
maxUnavailable1At least 4 Pods available: reduced capacity is tolerated
maxUnavailable: 0 with maxSurge: 1your choiceNo capacity ever lost, but one more Pod during the switch

A Pod is counted as available only when it is ready. A readiness probe (readinessProbe, see Probes) prevents a Pod that is still starting from receiving traffic and from making the switch move forward too early.

How do you detect a stuck rollout?

A rollout whose new Pods never become ready does not fail on its own: it waits. The progressDeadlineSeconds field (600 seconds by default) sets the delay after which Kubernetes declares it stuck. Reduce it to 20 seconds for the exercise, then ship an image that does not exist (# ❌, non-existent tag, deliberate):

Terminal window
kubectl patch deployment shop -p '{"spec":{"progressDeadlineSeconds":20}}'
kubectl set image deployment/shop nginx=nginx:1.30.999 # ❌ non-existent tag, deliberate
sleep 30
kubectl rollout status deployment/shop --timeout=10s
kubectl get deployment shop -o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}{"\n"}'
kubectl get pods -l app=shop
Output
deployment.apps/shop patched
deployment.apps/shop image updated
error: deployment "shop" exceeded its progress deadline
ProgressDeadlineExceeded
NAME READY STATUS RESTARTS AGE
shop-56f6b69df8-7v7fv 1/1 Running 0 33s
shop-56f6b69df8-b4kss 1/1 Running 0 32s
shop-56f6b69df8-ckcl4 1/1 Running 0 33s
shop-56f6b69df8-x92bj 1/1 Running 0 32s
shop-5b6779f6f5-8xzgt 0/1 ErrImagePull 0 30s
shop-5b6779f6f5-9nvmg 0/1 ImagePullBackOff 0 30s
shop-5b6779f6f5-p2ff7 0/1 ErrImagePull 0 30s

The exceeded its progress deadline line is an intended error: it signals that the rollout is stuck, and the Progressing condition carries the reason ProgressDeadlineExceeded. Kubernetes does not roll back on its own: it is up to you, or to your delivery tool.

Count the Pods: there are seven. Four are of the old version, the minimum guaranteed by maxUnavailable: 1, and keep serving. The other three are the new Pods, stuck on the image download: ErrImagePull when Kubernetes has just failed, ImagePullBackOff when it waits before retrying.

How do you roll back?

You have already undone an update with kubectl rollout undo in the lesson Deployments: the command reactivates the previous revision. Here, you add --to-revision, which targets a specific revision, and you watch how the numbers evolve. Start by reading the history:

Terminal window
kubectl rollout history deployment/shop
kubectl rollout undo deployment/shop
kubectl rollout status deployment/shop --timeout=90s
kubectl get pods -l app=shop
kubectl rollout history deployment/shop
Output
deployment.apps/shop
REVISION CHANGE-CAUSE
2 <none>
3 <none>
4 <none>
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/shop rolled back
Waiting for deployment "shop" rollout to finish: 4 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
NAME READY STATUS RESTARTS AGE
shop-56f6b69df8-7v7fv 1/1 Running 0 34s
shop-56f6b69df8-b4kss 1/1 Running 0 33s
shop-56f6b69df8-b5c4t 1/1 Running 0 1s
shop-56f6b69df8-ckcl4 1/1 Running 0 34s
shop-56f6b69df8-x92bj 1/1 Running 0 33s
deployment.apps/shop
REVISION CHANGE-CAUSE
2 <none>
4 <none>
5 <none>

Each undo renumbers the reactivated revision: it takes the highest number. Here, the history goes from 2, 3, 4 to 2, 4, 5: revision 3, the one for nginx 1.30.5, became 5, and revision 4, the one for the wrong image, is still listed.

Revision 1 appears in neither list: the ReplicaSet for nginx 1.30.5 was already reactivated by the kubectl apply of the previous section, and it took number 3 at that moment.

The Warning displayed by undo comes from the creation of the Deployment with kubectl apply. This command records the last applied configuration in the kubectl.kubernetes.io/last-applied-configuration annotation, and undo does not update it. In practice, a later kubectl apply of the original file would put the image of the file back, whether the annotation is up to date or not: kubectl apply only uses it to know which fields to remove.

To target version 1.30.6, read its number in the history. The command kubectl rollout history deployment/shop --revision=2 displays the Pod template of revision 2, image included: you thus check that it matches the desired version before going back to it. Asking for a revision that does not exist fails, as the first command shows (intended error):

Terminal window
kubectl rollout undo deployment/shop --to-revision=99
kubectl rollout history deployment/shop --revision=2
kubectl rollout undo deployment/shop --to-revision=2
kubectl rollout status deployment/shop --timeout=90s
kubectl get replicaset -l app=shop -o wide
Output
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
error: unable to find specified revision 99 in history
deployment.apps/shop with revision #2
Pod Template:
Labels: app=shop
pod-template-hash=685d5d7bc5
Containers:
nginx:
Image: nginx:1.30.6@sha256:42c68eab6f7e22611786a9fe1d1c019e275c50185bd54b8f75dc2b9184322703
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Node-Selectors: <none>
Tolerations: <none>
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/shop rolled back
Waiting for deployment spec update to be observed...
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
shop-56f6b69df8 0 0 0 72s nginx nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb app=shop,pod-template-hash=56f6b69df8
shop-5b6779f6f5 0 0 0 33s nginx nginx:1.30.999 app=shop,pod-template-hash=5b6779f6f5
shop-685d5d7bc5 5 5 5 55s nginx nginx:1.30.6@sha256:42c68eab6f7e22611786a9fe1d1c019e275c50185bd54b8f75dc2b9184322703 app=shop,pod-template-hash=685d5d7bc5

The undo to revision 99 fails without changing anything, the one to revision 2 succeeds. At the end, the ReplicaSet for nginx 1.30.6 carries the five Pods again, and the other two, including the one for the wrong image, are at zero.

Annotate the changes

A revision without a cause does not say why it exists. You have already filled the CHANGE-CAUSE column with the kubernetes.io/change-cause annotation in the lesson Deployments. Here is the action on this Deployment, then the trap it hides:

Terminal window
kubectl annotate deployment/shop kubernetes.io/change-cause="retour à nginx 1.30.6" --overwrite
kubectl rollout history deployment/shop
Output
deployment.apps/shop annotated
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6

The --overwrite option is needed when the annotation already exists; without it, kubectl annotate refuses to replace it.

The trap is that this annotation lives on the Deployment, not on the revision: the next rollout copies it as is, and the history starts to lie. The next section shows it. Remember the rule: rewrite the annotation at every delivery, never just once. The full syntax of kubectl annotate is in Labels and annotations.

How do you group several changes into a single rollout?

Every change to the Pod template triggers a rollout. To group several changes without triggering one rollout per change, pause the Deployment, make your changes, then resume: the Deployment applies the final template at once. Two new changes serve for the demonstration: an environment variable and a CPU request.

Terminal window
kubectl rollout pause deployment/shop
kubectl set env deployment/shop MODE=pause-demo
kubectl set resources deployment/shop --requests=cpu=50m
kubectl get replicaset -l app=shop
kubectl rollout history deployment/shop
Output
deployment.apps/shop paused
deployment.apps/shop env updated
deployment.apps/shop resource requirements updated
NAME DESIRED CURRENT READY AGE
shop-56f6b69df8 0 0 0 72s
shop-5b6779f6f5 0 0 0 33s
shop-685d5d7bc5 5 5 5 55s
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6

Look at the two lists after the set commands. There are still three ReplicaSets, and the last one, the one for nginx 1.30.6, keeps its five Pods. The history stops at revision 6: during the pause, the two set commands modify the Deployment without anything moving. Now resume the rollout:

Terminal window
kubectl rollout resume deployment/shop
kubectl rollout status deployment/shop --timeout=90s
kubectl get replicaset -l app=shop
kubectl rollout history deployment/shop
Output
deployment.apps/shop resumed
Waiting for deployment spec update to be observed...
Waiting for deployment "shop" rollout to finish: 2 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 1 old replicas are pending termination...
[...]
deployment "shop" successfully rolled out
NAME DESIRED CURRENT READY AGE
shop-56f6b69df8 0 0 0 74s
shop-5b6779f6f5 0 0 0 35s
shop-685d5d7bc5 0 0 0 57s
shop-78c9895f97 5 5 5 2s
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6
7 retour à nginx 1.30.6

On resume, a single rollout takes place: a single new ReplicaSet (shop-78c9895f97) carries the five Pods, and the history gains a single revision, number 7, which carries both MODE=pause-demo and the CPU request. Without the pause, each of the two changes would have triggered its own rollout and its own revision.

Revision 7 nevertheless displays the cause of revision 6, "retour à nginx 1.30.6", although its template changed: this is the trap described above, and the fix is to rewrite the annotation:

Terminal window
kubectl annotate deployment/shop kubernetes.io/change-cause="MODE=pause-demo et cpu 50m" --overwrite
kubectl rollout history deployment/shop
Output
deployment.apps/shop annotated
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6
7 MODE=pause-demo et cpu 50m

Only the last line changed: revision 7 now carries its real cause, and the previous ones keep theirs.

What should you do when an update does not finish?

Start from the symptom of kubectl rollout status and go down one level at a time.

SymptomWhere to lookFrequent cause
exceeded its progress deadlinekubectl get pods -l app=<name>Image not found, failing readiness probe
New Pods in ImagePullBackOffkubectl describe pod <name-of-a-shop-pod>Mistake in the image or private registry
New Pods Running but never Readykubectl describe pod <name-of-a-shop-pod>Failing readiness probe
Very slow switchkubectl get deployment shop -o jsonpath='{.spec.strategy}'maxSurge too low, or high minReadySeconds (the delay during which a ready Pod must stay ready before being counted as available)

Blue-green and canary, which change the traffic rather than the Pods, belong to the Service: see Services and Deployment strategies.

Clean up the cluster

Delete the shop Deployment to start again from a clean cluster. The list of Pods, taken right after, shows them at different stages of their shutdown: some Running or Terminating, others already Completed.

Terminal window
kubectl delete deployment shop
kubectl get pods -l app=shop
Output
deployment.apps "shop" deleted from default namespace
NAME READY STATUS RESTARTS AGE
shop-685d5d7bc5-fm6zs 0/1 Completed 0 4s
shop-685d5d7bc5-lgk6l 0/1 Completed 0 3s
shop-78c9895f97-55w7j 1/1 Terminating 0 2s
shop-78c9895f97-5vvxl 1/1 Terminating 0 1s
shop-78c9895f97-7rbbp 1/1 Running 0 2s
shop-78c9895f97-qrffg 1/1 Running 0 2s
shop-78c9895f97-tstds 1/1 Terminating 0 1s

Put it into practice

The lab starts from a Deployment of five replicas set to Recreate. You switch it to RollingUpdate with maxSurge at 2 and maxUnavailable at 1, then you ship the nginx 1.30.6 image. The validation reads the state of the cluster: the strategy and its two bounds, the image of the Deployment, the five ready replicas on the new ReplicaSet, and the old ReplicaSets kept at zero.

Key takeaways

  1. A rolling update replaces the Pods in steps, bounded by maxSurge and maxUnavailable (25% by default).
  2. With 5 replicas, maxSurge: 2 and maxUnavailable: 1, there are at most 7 Pods and at least 4 available.
  3. Recreate cuts the application but guarantees that only one version runs; RollingUpdate avoids downtime as long as the new Pods become ready.
  4. progressDeadlineSeconds declares a rollout stuck, but Kubernetes does not roll back on its own.
  5. kubectl rollout undo reactivates the previous revision, and --to-revision a specific one; the reactivated revision takes the highest number.
  6. kubernetes.io/change-cause is rewritten at every delivery, with --overwrite.
  7. kubectl rollout pause then resume groups several changes into a single rollout.

Test your knowledge

This quiz checks that you can predict the number of Pods during a rolling update, tell Recreate from RollingUpdate, and read the revision numbers after an undo. Answer without going back: what blocks you points to the passage to reread.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer