Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Level 4 exam: administer a cluster

By updated

Improve this page

This exam measures whether you can, on your own, take over a kubeadm cluster at a moment's notice: bring a worker back, prove that a control plane survives the loss of one of its members, back up etcd, restore a deleted shop, upgrade the cluster by one minor version and repair three failures. It is meant for people who are finishing level 4, "Administer and operate a cluster". It runs on three virtual machines of the lab catalogue, k8s-cp.lab, k8s-w1.lab and k8s-kind.lab, so plan about 180 minutes. You pass it by meeting the seven critical criteria and reaching at least 70 points out of 100. It offers four hints, which cost between 10 and 25 points each, and it teaches nothing new: it checks what the lessons have taught.

What you will learn

  • Say what the exam expects, and on which machines it grades you.
  • Spot the level lessons to review before you start.
  • Work out what you must pass: the seven critical criteria, then enough others to reach 70 points.
  • Know what a hint costs, and how to take one, to decide whether to ask for one.

What should you be able to do before you start?

No step-by-step guide goes with the exam, and choosing the right steps is part of what it measures. The specification names no Kubernetes object: it describes results. Everything it asks is taught in the Installing a cluster, Administering the cluster and Day-to-day operations modules (Diagnosing, Maintaining and upgrading, Making reliable and operating), or in the earlier levels, whose exams already checked the deployment of an application. If a step feels unclear, reread those modules before you start.

The exam also assumes a troubleshooting reflex: three failures are shipped with no explanation, and the cause of each must be written before the repair. The section "What should you do after the exam?" lists the lessons tied to each criterion.

What does the specification ask for?

The scenario has you take over the kubeadm cluster that hosts a shop. It says what must be true at the end, never how to get there. The lab scenario gives the names that the grader looks for: it is authoritative, so read it in full when you start.

You start on the control plane k8s-cp.lab, with kubectl configured and sudo rights. The machines k8s-w1.lab and k8s-kind.lab are reached with ssh, also with sudo. On k8s-kind.lab, Docker and kind are installed: this is where you build, for the high availability proof, a kind cluster named ha, with the node image from the catalogue, kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5.

The inherited state has seven problems.

  • The control plane is in a mess: one node is NotReady and kubectl no longer answers.
  • The worker k8s-w1.lab was reset and is no longer in the cluster.
  • The whole shop was deleted the evening before. An etcd backup exists in /var/backups/etcd/etcd-veille.db, and an object named apres-la-veille has been written since.
  • The shop's manifests live in the git repository ~/epreuve/manifestes, and some of them use removed APIs.
  • The cluster runs 1.36.4 and must move to 1.37.0.
  • Two control plane certificates were reissued for a few days only.
  • The panier component does not start once the shop is back.

The specification does not say how to settle these problems. It comes down to eighteen requirements, each graded by one criterion.

  • The worker k8s-w1.lab is part of the cluster again, Ready, and its kubelet reports a 1.37 version.
  • Swap is off on the worker, and nothing will turn it back on at the next boot.
  • The ha cluster has three control planes, all Ready, read through the address of the load balancer placed in front of their API servers.
  • The API of ha still answers through that address when one of its control planes is stopped.
  • Before restoring anything, a fresh etcd backup is taken to /var/backups/etcd/etcd-avant-restauration.db, and verified.
  • The shop came back by restoring the backup from the evening before: these are the same objects, not copies recreated from the repository.
  • The apres-la-veille object no longer exists, and all the nodes are Ready.
  • At the last commit of the repository, no manifest uses an API removed from Kubernetes any more, and the API server accepts them all. None disappears from the repository.
  • The API server and every node report 1.37.0, no node was left out of scheduling, and the shop's storefront keeps all its replicas ready.
  • During any voluntary maintenance, the storefront keeps at least one replica available, and the cluster guarantees it.
  • The worker was drained without taking the shop down, then put back in service: it accepts Pods again.
  • The NotReady node is Ready, and its kubelet will start again by itself after a reboot.
  • The API server answers again and still authorizes requests through the Node and RBAC modes.
  • panier has all its replicas available, no longer restarts, and reads its configuration where it looks for it.
  • Before each repair, the cause is recorded in ~/epreuve/constats/, based on describe, the events or the logs: a kubelet.txt file, an apiserver.txt file and a panier.txt file, each naming what was at fault. Afterwards, they are not touched again: their date is compared with the start time of whatever was repaired.
  • What you changed live on panier carries the same value at the last commit of the repository, with no gap from the cluster.
  • No control plane certificate expires in less than thirty days.
  • The storefront's replicas run on different nodes, and none receives traffic before it is ready to serve.

The order of work matters: a restore brings the whole cluster back to the evening before, including what you did to it before the restore.

How do you start the exam and get graded?

The lab prepares the machines and the full statement, then reads the state of both clusters, of the repository and of the findings again when you ask for your grade. You start it with dsoxlab run k8s-epreuve-administrer-un-cluster. Three commands are then used: dsoxlab hint k8s-epreuve-administrer-un-cluster gives a paid hint, dsoxlab check k8s-epreuve-administrer-un-cluster tells you where you stand as many times as you like, and dsoxlab submit k8s-epreuve-administrer-un-cluster returns the verdict. The exams of the earlier levels had no hints: this one does, and each one reduces your scale. The grade never looks at the commands you typed, only at the final state.

Each check acts on the clusters. The grader itself pauses a control plane of the ha cluster, then starts it again, to check that the API answers during the outage. It also launches an ephemeral Pod in the shop, sonde-placement, which the scheduler must place on the worker. What comes back is limited to the verdict, passed or not, and the grade.

Put it into practice

This lab is the exam itself. It puts you in front of the scenario described above and it is validated on the final state of the clusters, of the repository and of the findings, as a colleague taking over your work would.

How is the exam graded?

Eighteen criteria make up the grade, at equal weight: each is worth about 5.6 points out of 100. The pass mark is 70 points. Seven criteria are critical: they must all pass, and failing a single one fails the exam, whatever the total. The seven critical criteria are worth about 38.9 points: with no hint, you must therefore pass all of them, then at least six of the other eleven to reach 70, that is thirteen criteria in all.

There are four hints, from the vaguest to the most precise. They cost 10, 15, 20 and 25 points. This cost is taken off the scale before the grade is computed: the grade is the share of criteria passed, applied to that reduced scale. A hint taken makes the pass mark harder to reach: with the first one, at 10 points, the scale drops to 90 and you need fourteen criteria instead of thirteen. Taking all four removes 70 points, and the maximum falls to 30. As soon as the hints taken cost more than 30 points in total, 70 points can no longer be reached, even with all eighteen criteria. So take a hint to get unstuck, never as a shortcut.

CriterionCritical
P01the reset worker joins the cluster, it is Ready and its kubelet reports version 1.37yes
P02swap is disabled on the worker and stays so after a rebootno
P03the three control plane nodes are Ready, read through the load balancer addressyes
P04the API answers through the load balancer address while a control plane is stoppedyes
P05a fresh etcd backup exists, it is verified and newer than the one from the evening beforeno
P06the deleted shop came back with its original identifiers after a restore from the backup of the evening beforeyes
P07an object created after the backup of the evening before no longer exists and the nodes are Readyno
P08the manifests on a removed API are migrated to the current API and applyno
P09the API server and every node report the target version, no node is unschedulable and the shop keeps its replicas readyyes
P10during a voluntary maintenance, the storefront keeps at least one replica available, and the cluster guarantees itno
P11the worker is drained without taking the shop down, then put back in service, and it accepts Pods againno
P12the NotReady node is Ready and its kubelet will survive a rebootyes
P13the API server answers again and still authorizes requests through the Node and RBAC modesyes
P14panier has all its replicas available, no longer restarts and reads its configuration where it looks for itno
P15the cause of each failure is recorded, based on describe, the events or the logs, before the repairno
P16a field changed live during the incident carries the same value in the versioned manifestno
P17no control plane certificate expires in less than thirty daysno
P18the storefront's replicas run on different nodes, and none receives traffic before it is ready to serveno

What should you do after the exam?

If the seven critical criteria pass and the total reaches 70, the level is validated. Otherwise, reread the lesson that teaches the failed criterion, then retake the exam:

Key points

  • The exam is a measurement: it explains nothing again and grades the final state of the clusters, of the repository and of the findings, not your commands.
  • The seven critical criteria decide the verdict: failing one fails the exam, and the total must also reach 70 points.
  • Order matters: a restore brings the cluster back to the evening before, including what you did before it.
  • The cause of each failure is written in ~/epreuve/constats/ before the repair, and the files are not touched again afterwards.
  • A hint costs 10, 15, 20 or 25 points depending on its precision, and beyond 30 points of hints in total the pass mark is out of reach; dsoxlab check tells you where you stand at will and only dsoxlab submit returns the verdict.