Kubernetes rolling updates and rollbacks: tune and undo
Tune maxSurge and maxUnavailable, switch from Recreate to RollingUpdate, detect a stuck rollout with progressDeadlineSeconds and roll back a Deployment.
The issue opens pre-filled with this page title and URL.
A rolling update replaces the Pods of a Deployment in small groups: the new ones start while the old ones still serve. This lesson shows how to tune the speed of this replacement (maxSurge, maxUnavailable), how to switch from the Recreate strategy to RollingUpdate, how to detect a stuck rollout with progressDeadlineSeconds, and how to read, annotate and undo the revision history. You play it all on a Deployment of five replicas.
Prerequisites: the lesson Deployments, which creates a Deployment, ships a first update and undoes a wrong image, and the demo cluster from First cluster. This lesson does not repeat those actions: it tunes and goes deeper.
What you will learn
Trigger an update with kubectl set image or by changing the manifest, and follow it with kubectl rollout status
Tune maxSurge and maxUnavailable, and predict the number of Pods during the switch
Switch from Recreate to RollingUpdate and read the difference in the events
Detect a stuck rollout with progressDeadlineSeconds and go back with kubectl rollout undo
Read and annotate the revision history (change-cause)
Group several changes into a single rollout with the pause
How does a rolling update work?
At every change to the Pod template, the Deployment creates a new ReplicaSet. It then increases the number of Pods of the new ReplicaSet and decreases that of the old one, in steps, as long as two guardrails are respected: maxSurge (how many Pods above the desired number are tolerated) and maxUnavailable (how many unavailable Pods are tolerated). Both are 25% by default.
The diagram shows the example of three Pods: at each step a Pod of version 2 appears before a Pod of version 1 disappears, and the old ReplicaSet stays at zero at the end, ready for a rollback. Only a change to the Pod template (image, environment variable, template label) triggers this mechanism: changing replicas creates no ReplicaSet.
The calculation is simple. With 5 replicas, maxSurge: 2 and maxUnavailable: 1, there will be at any moment at most 5 + 2 = 7 Pods, and at least 5 - 1 = 4 Pods available. A value can be written as a number or as a percentage of the number of replicas.
How does the Recreate strategy behave?
The Recreate strategy deletes all the old Pods before creating the new ones: the application is cut for a while, but two versions never run together. Create the shop Deployment of five replicas with this strategy, then update it:
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
deployment.apps/shop image updated
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 0 of 5 updated replicas are available...
[...]
deployment "shop" successfully rolled out
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 36s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 5
Normal ScalingReplicaSet 20s deployment-controller Scaled down replica set shop-56f6b69df8 from 5 to 0
Normal ScalingReplicaSet 19s deployment-controller Scaled up replica set shop-685d5d7bc5 from 0 to 5
Read the order of the events: the old ReplicaSet goes down from 5 to 0, and only then does the new one go up from 0 to 5. During the interval, no Pod serves: this is the price of the Recreate strategy, which suits cases where two versions cannot coexist (for example because of a database schema).
How do you switch to RollingUpdate with maxSurge and maxUnavailable?
Now switch to RollingUpdate, with maxSurge: 2 and maxUnavailable: 1, by changing the manifest. So that this change also ships a new version, the manifest goes back to the starting image:
Normal ScalingReplicaSet 39s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 5
Normal ScalingReplicaSet 23s deployment-controller Scaled down replica set shop-56f6b69df8 from 5 to 0
Normal ScalingReplicaSet 22s deployment-controller Scaled up replica set shop-685d5d7bc5 from 0 to 5
Normal ScalingReplicaSet 3s deployment-controller Scaled up replica set shop-56f6b69df8 from 0 to 2
Normal ScalingReplicaSet 3s deployment-controller Scaled down replica set shop-685d5d7bc5 from 5 to 4
Normal ScalingReplicaSet 2s deployment-controller Scaled up replica set shop-56f6b69df8 from 2 to 3
Normal ScalingReplicaSet 2s deployment-controller Scaled down replica set shop-685d5d7bc5 from 4 to 3
Normal ScalingReplicaSet 2s deployment-controller Scaled up replica set shop-56f6b69df8 from 3 to 4
Normal ScalingReplicaSet 1s deployment-controller Scaled down replica set shop-685d5d7bc5 from 3 to 2
Normal ScalingReplicaSet 1s (x3 over 1s) deployment-controller (combined from similar events): Scaled down replica set shop-685d5d7bc5 from 1 to 0
The first three events are those of the previous step. The following ones alternate between the two ReplicaSets: the new one goes from 0 to 2 while the old one goes from 5 to 4, then from 2 to 3 while the old one goes to 3, and so on down to zero. Between two steps, the Deployment stays within its bounds: 7 Pods at most, 4 available at least.
Also notice that there are only two ReplicaSets at the end, although the image took three successive values (1.30.5, 1.30.6, then 1.30.5 again). The kubectl applyreactivated the ReplicaSet of image 1.30.5, created at the start: a ReplicaSet whose Pod template is identical is reused, not recreated. The settings read as follows:
Setting
Value in the example
Effect
maxSurge
2
Up to 7 Pods for 5 desired: the replacement goes faster, at the cost of extra resources
maxUnavailable
1
At least 4 Pods available: reduced capacity is tolerated
maxUnavailable: 0 with maxSurge: 1
your choice
No capacity ever lost, but one more Pod during the switch
Swipe to see more
A Pod is counted as available only when it is ready. A readiness probe (readinessProbe, see Probes) prevents a Pod that is still starting from receiving traffic and from making the switch move forward too early.
How do you detect a stuck rollout?
A rollout whose new Pods never become ready does not fail on its own: it waits. The progressDeadlineSeconds field (600 seconds by default) sets the delay after which Kubernetes declares it stuck. Reduce it to 20 seconds for the exercise, then ship an image that does not exist (# ❌, non-existent tag, deliberate):
error: deployment "shop" exceeded its progress deadline
ProgressDeadlineExceeded
NAME READY STATUS RESTARTS AGE
shop-56f6b69df8-7v7fv 1/1 Running 0 33s
shop-56f6b69df8-b4kss 1/1 Running 0 32s
shop-56f6b69df8-ckcl4 1/1 Running 0 33s
shop-56f6b69df8-x92bj 1/1 Running 0 32s
shop-5b6779f6f5-8xzgt 0/1 ErrImagePull 0 30s
shop-5b6779f6f5-9nvmg 0/1 ImagePullBackOff 0 30s
shop-5b6779f6f5-p2ff7 0/1 ErrImagePull 0 30s
The exceeded its progress deadline line is an intended error: it signals that the rollout is stuck, and the Progressing condition carries the reason ProgressDeadlineExceeded. Kubernetes does not roll back on its own: it is up to you, or to your delivery tool.
Count the Pods: there are seven. Four are of the old version, the minimum guaranteed by maxUnavailable: 1, and keep serving. The other three are the new Pods, stuck on the image download: ErrImagePull when Kubernetes has just failed, ImagePullBackOff when it waits before retrying.
How do you roll back?
You have already undone an update with kubectl rollout undo in the lesson Deployments: the command reactivates the previous revision. Here, you add --to-revision, which targets a specific revision, and you watch how the numbers evolve. Start by reading the history:
Terminal window
kubectlrollouthistorydeployment/shop
kubectlrolloutundodeployment/shop
kubectlrolloutstatusdeployment/shop--timeout=90s
kubectlgetpods-lapp=shop
kubectlrollouthistorydeployment/shop
Output
deployment.apps/shop
REVISION CHANGE-CAUSE
2 <none>
3 <none>
4 <none>
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/shop rolled back
Waiting for deployment "shop" rollout to finish: 4 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
NAME READY STATUS RESTARTS AGE
shop-56f6b69df8-7v7fv 1/1 Running 0 34s
shop-56f6b69df8-b4kss 1/1 Running 0 33s
shop-56f6b69df8-b5c4t 1/1 Running 0 1s
shop-56f6b69df8-ckcl4 1/1 Running 0 34s
shop-56f6b69df8-x92bj 1/1 Running 0 33s
deployment.apps/shop
REVISION CHANGE-CAUSE
2 <none>
4 <none>
5 <none>
Each undorenumbers the reactivated revision: it takes the highest number. Here, the history goes from 2, 3, 4 to 2, 4, 5: revision 3, the one for nginx 1.30.5, became 5, and revision 4, the one for the wrong image, is still listed.
Revision 1 appears in neither list: the ReplicaSet for nginx 1.30.5 was already reactivated by the kubectl apply of the previous section, and it took number 3 at that moment.
The Warning displayed by undo comes from the creation of the Deployment with kubectl apply. This command records the last applied configuration in the kubectl.kubernetes.io/last-applied-configuration annotation, and undo does not update it. In practice, a later kubectl apply of the original file would put the image of the file back, whether the annotation is up to date or not: kubectl apply only uses it to know which fields to remove.
To target version 1.30.6, read its number in the history. The command kubectl rollout history deployment/shop --revision=2 displays the Pod template of revision 2, image included: you thus check that it matches the desired version before going back to it. Asking for a revision that does not exist fails, as the first command shows (intended error):
Terminal window
kubectlrolloutundodeployment/shop--to-revision=99
kubectlrollouthistorydeployment/shop--revision=2
kubectlrolloutundodeployment/shop--to-revision=2
kubectlrolloutstatusdeployment/shop--timeout=90s
kubectlgetreplicaset-lapp=shop-owide
Output
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
error: unable to find specified revision 99 in history
Warning: resource deployments/shop was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/shop rolled back
Waiting for deployment spec update to be observed...
Waiting for deployment "shop" rollout to finish: 0 out of 5 new replicas have been updated...
[...]
deployment "shop" successfully rolled out
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
The undo to revision 99 fails without changing anything, the one to revision 2 succeeds. At the end, the ReplicaSet for nginx 1.30.6 carries the five Pods again, and the other two, including the one for the wrong image, are at zero.
Annotate the changes
A revision without a cause does not say why it exists. You have already filled the CHANGE-CAUSE column with the kubernetes.io/change-cause annotation in the lesson Deployments. Here is the action on this Deployment, then the trap it hides:
Terminal window
kubectlannotatedeployment/shopkubernetes.io/change-cause="retour à nginx 1.30.6"--overwrite
kubectlrollouthistorydeployment/shop
Output
deployment.apps/shop annotated
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6
The --overwrite option is needed when the annotation already exists; without it, kubectl annotate refuses to replace it.
The trap is that this annotation lives on the Deployment, not on the revision: the next rollout copies it as is, and the history starts to lie. The next section shows it. Remember the rule: rewrite the annotation at every delivery, never just once. The full syntax of kubectl annotate is in Labels and annotations.
How do you group several changes into a single rollout?
Every change to the Pod template triggers a rollout. To group several changes without triggering one rollout per change, pause the Deployment, make your changes, then resume: the Deployment applies the final template at once. Two new changes serve for the demonstration: an environment variable and a CPU request.
Look at the two lists after the set commands. There are still three ReplicaSets, and the last one, the one for nginx 1.30.6, keeps its five Pods. The history stops at revision 6: during the pause, the two set commands modify the Deployment without anything moving. Now resume the rollout:
Terminal window
kubectlrolloutresumedeployment/shop
kubectlrolloutstatusdeployment/shop--timeout=90s
kubectlgetreplicaset-lapp=shop
kubectlrollouthistorydeployment/shop
Output
deployment.apps/shop resumed
Waiting for deployment spec update to be observed...
Waiting for deployment "shop" rollout to finish: 2 out of 5 new replicas have been updated...
[...]
Waiting for deployment "shop" rollout to finish: 1 old replicas are pending termination...
[...]
deployment "shop" successfully rolled out
NAME DESIRED CURRENT READY AGE
shop-56f6b69df8 0 0 0 74s
shop-5b6779f6f5 0 0 0 35s
shop-685d5d7bc5 0 0 0 57s
shop-78c9895f97 5 5 5 2s
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6
7 retour à nginx 1.30.6
On resume, a single rollout takes place: a single new ReplicaSet (shop-78c9895f97) carries the five Pods, and the history gains a single revision, number 7, which carries both MODE=pause-demo and the CPU request. Without the pause, each of the two changes would have triggered its own rollout and its own revision.
Revision 7 nevertheless displays the cause of revision 6, "retour à nginx 1.30.6", although its template changed: this is the trap described above, and the fix is to rewrite the annotation:
Terminal window
kubectlannotatedeployment/shopkubernetes.io/change-cause="MODE=pause-demo et cpu 50m"--overwrite
kubectlrollouthistorydeployment/shop
Output
deployment.apps/shop annotated
deployment.apps/shop
REVISION CHANGE-CAUSE
4 <none>
5 <none>
6 retour à nginx 1.30.6
7 MODE=pause-demo et cpu 50m
Only the last line changed: revision 7 now carries its real cause, and the previous ones keep theirs.
What should you do when an update does not finish?
Start from the symptom of kubectl rollout status and go down one level at a time.
Symptom
Where to look
Frequent cause
exceeded its progress deadline
kubectl get pods -l app=<name>
Image not found, failing readiness probe
New Pods in ImagePullBackOff
kubectl describe pod <name-of-a-shop-pod>
Mistake in the image or private registry
New Pods Running but never Ready
kubectl describe pod <name-of-a-shop-pod>
Failing readiness probe
Very slow switch
kubectl get deployment shop -o jsonpath='{.spec.strategy}'
maxSurge too low, or high minReadySeconds (the delay during which a ready Pod must stay ready before being counted as available)
Swipe to see more
Blue-green and canary, which change the traffic rather than the Pods, belong to the Service: see Services and Deployment strategies.
Clean up the cluster
Delete the shop Deployment to start again from a clean cluster. The list of Pods, taken right after, shows them at different stages of their shutdown: some Running or Terminating, others already Completed.
Terminal window
kubectldeletedeploymentshop
kubectlgetpods-lapp=shop
Output
deployment.apps "shop" deleted from default namespace
NAME READY STATUS RESTARTS AGE
shop-685d5d7bc5-fm6zs 0/1 Completed 0 4s
shop-685d5d7bc5-lgk6l 0/1 Completed 0 3s
shop-78c9895f97-55w7j 1/1 Terminating 0 2s
shop-78c9895f97-5vvxl 1/1 Terminating 0 1s
shop-78c9895f97-7rbbp 1/1 Running 0 2s
shop-78c9895f97-qrffg 1/1 Running 0 2s
shop-78c9895f97-tstds 1/1 Terminating 0 1s
Put it into practice
The lab starts from a Deployment of five replicas set to Recreate. You switch it to RollingUpdate with maxSurge at 2 and maxUnavailable at 1, then you ship the nginx 1.30.6 image. The validation reads the state of the cluster: the strategy and its two bounds, the image of the Deployment, the five ready replicas on the new ReplicaSet, and the old ReplicaSets kept at zero.
Key takeaways
A rolling update replaces the Pods in steps, bounded by maxSurge and maxUnavailable (25% by default).
With 5 replicas, maxSurge: 2 and maxUnavailable: 1, there are at most 7 Pods and at least 4 available.
Recreate cuts the application but guarantees that only one version runs; RollingUpdate avoids downtime as long as the new Pods become ready.
progressDeadlineSeconds declares a rollout stuck, but Kubernetes does not roll back on its own.
kubectl rollout undo reactivates the previous revision, and --to-revision a specific one; the reactivated revision takes the highest number.
kubernetes.io/change-cause is rewritten at every delivery, with --overwrite.
kubectl rollout pause then resume groups several changes into a single rollout.
Test your knowledge
This quiz checks that you can predict the number of Pods during a rolling update, tell Recreate from RollingUpdate, and read the revision numbers after an undo. Answer without going back: what blocks you points to the passage to reread.
Knowledge check
Check what you have learned with this interactive quiz
7 questions
5 min
80% to pass
How it works
The timer starts when you click Start
Multiple choice, true/false and short answer questions
You can move between questions
Detailed results are shown at the end
Starts the quiz and the timer
⏱️--:--
0 / 7
Review
(0/0)
--:--
0%
0Correct
0To review
0:00Time
-Level
Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.