Getting your work checked with dsoxlab check and submit

dsoxlab check plays the lab's tests, computes the mark, records it and displays a state per criterion: satisfied, unsatisfied, not evaluated or skipped. What it says about a failed criterion depends on the lab's type: the author's full message on a guided lab, a short hint on a challenge, the verdict alone on a mock exam (a lab that sets a pass mark). This lesson is for anyone who already plays labs and wants to read a result without guessing. It compares check and submit, explains what a mock exam judges, and shows how to read a lab's state with status.

What you will learn

• Read the result of dsoxlab check criterion by criterion.
• Tell an unsatisfied criterion from a criterion that was not evaluated, which is not your fault.
• Know what check shows of a failure on a guided lab, a challenge and a mock exam.
• Compare check and submit, and read the verdict of a mock exam.
• Read the state of a lab with dsoxlab status.

Prerequisites

You need the dsoxlab CLI in version 0.8.0 or newer, a lab started as the lesson Starting a dsoxlab lab with dsoxlab start describes, and the habit of the loop shown by the lesson Playing your first dsoxlab lab with dsoxlab demo.

What does dsoxlab check do, and what does dsoxlab submit change?

dsoxlab check <id> runs the lab's tests, computes the mark by deducting the cost of the hints, records it, and displays the detail of each criterion. You can run it again as many times as you like: each pass records an attempt, failed attempts included, and the best one becomes the score that show and list-labs display.

The score is the proportion of tests passed, applied to a scale of 100 points reduced by the cost of the hints taken, and it never falls below zero.

dsoxlab check l2-swap-management
dsoxlab check drill-storage --target ubuntu   # multi-host lab of the linux catalogue: we test the ubuntu target
dsoxlab submit l2-swap-management             # same validation, then close the session

dsoxlab submit does the same validation then closes the session: in the subshell opened by run, you then have to type exit, and submit also empties the active lab. On an ordinary lab, that is the only difference: you play check while you fix, and submit when you are done. On a lab that declares a pass mark, a mock exam, submit also returns a verdict.

The exit codes of check say the same thing as the screen. It exits with 1 when a test fails, which concerns your work, and with 2 when the infrastructure of a vm lab is not provisioned, which concerns your installation and records nothing.

How do you read the result of check, criterion by criterion?

Each test becomes a criterion, and a criterion has four states: satisfied, unsatisfied, not evaluated or skipped. Each one is read on screen during the run, and a summary "Ce qui reste à faire" (What is left to do) follows as soon as a criterion is not satisfied. Here is a first check, played with 0.8.0 on the docker-premier-conteneur lab just after dsoxlab run, before having done anything. The outputs of this lesson were captured with DSOXLAB_LANG=fr, so the messages are in French: the same messages exist in English.

$ dsoxlab check docker-premier-conteneur
ℹ Validation de docker-premier-conteneur…
  ✘ test_functional.py::test_un_conteneur_a_ete_lance_et_tourne  (FAILED)
  ⊘ test_functional.py::test_les_journaux_du_conteneur_portent_la_ligne_demandee
(skipped)
  ✘ test_functional.py::test_le_conteneur_en_service_a_ete_arrete  (FAILED)
  ✘ test_functional.py::test_le_conteneur_au_rebut_a_ete_supprime  (FAILED)
  Tests docker-premier-conteneur terminés ━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:01
Ce qui reste à faire, d'après les tests :
  ✘ un conteneur a ete lance et tourne
      aucun conteneur nommé `premier` n'existe. L'image `lab-base:1.0` est déjà
      sur la machine : rien à tirer, rien à construire.
      conteneurs présents : rebut, vitrine
  ✘ le conteneur en service a ete arrete
      `vitrine` tourne toujours. L'arrêter met fin à son processus tout en
      conservant le conteneur, ses journaux et son nom.
  ✘ le conteneur au rebut a ete supprime
      `rebut` existe toujours (état : exited). Arrêter un conteneur ne le
      supprime pas : tant qu'il existe, il garde son nom, et aucun autre
      conteneur ne peut le reprendre.
ℹ La sortie complète des tests : dsoxlab -v check docker-premier-conteneur
╭─ Résultat : docker-premier-conteneur ─╮
│  0 / 100 pts                          │
│  ░░░░░░░░░░░░░░░░░░░░  0/4 tests      │
╰───────────────────────────────────────╯
ℹ Résultat enregistré dans l'historique (0/100 pts).
✘ Des tests ont échoué.

Look first at the four lines at the top: a tick for a satisfied criterion, a cross for an unsatisfied criterion, and ⊘ for a skipped test. The logs test is skipped because it assumes the premier container, which does not exist yet: the test starts by looking for that container and skips itself if it is absent (this is the skip of the catalogue's test_functional.py file). A skipped test is neither passed nor failed, and the criterion does not appear in "Ce qui reste à faire".

It nevertheless counts as zero in the mark: dsoxlab check docker-premier-conteneur --json returns skipped: 1 and score: 0 on this fresh lab, and the command exits with code 1.

Then look at the three blocks at the bottom: each unsatisfied criterion carries the author's message, which says what the test observed and not only that it failed. Finally, the line "Résultat enregistré dans l'historique (0/100 pts)" (Result saved to history) confirms that a zero result is recorded too.

After starting the premier container (which prints premier-est-la) and stopping vitrine, without removing rebut, the second pass gives this:

$ dsoxlab check docker-premier-conteneur
ℹ Validation de docker-premier-conteneur…
  ✔ test_functional.py::test_un_conteneur_a_ete_lance_et_tourne
  ✔ test_functional.py::test_les_journaux_du_conteneur_portent_la_ligne_demandee
  ✔ test_functional.py::test_le_conteneur_en_service_a_ete_arrete
  ✘ test_functional.py::test_le_conteneur_au_rebut_a_ete_supprime  (FAILED)
  Tests docker-premier-conteneur terminés ━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:01
Ce qui reste à faire, d'après les tests :
  ✘ le conteneur au rebut a ete supprime
      `rebut` existe toujours (état : exited). Arrêter un conteneur ne le
      supprime pas : tant qu'il existe, il garde son nom, et aucun autre
      conteneur ne peut le reprendre.
ℹ La sortie complète des tests : dsoxlab -v check docker-premier-conteneur
╭─ Résultat : docker-premier-conteneur ─╮
│  75 / 100 pts                         │
│  ███████████████░░░░░  3/4 tests      │
╰───────────────────────────────────────╯
ℹ Résultat enregistré dans l'historique (75/100 pts).
✘ Des tests ont échoué.

The logs test went from skipped to satisfied as soon as premier exists, and only one criterion is left to fix. The score is 75, that is three tests out of four on a scale of 100, with no hint, and check still exits with code 1 as long as a test fails. When the four criteria are satisfied, the last line will become "Tous les tests sont passés" (All tests passed).

The fourth state reads differently. A criterion is not evaluated when the setup could not verify it: an error while preparing a test (a fixture, that is, the set-up the test expects) or a collection error. During the run, the line carries (ERROR). In "Ce qui reste à faire", it is displayed with a question mark and the words "non évalué : vérification impossible" (not evaluated: verification impossible). This is not your fault: nothing was measured on that criterion. A reasonable practice is to run check again, then, if the error comes back, to run dsoxlab doctor and report the problem with dsoxlab support.

State	What you see	Who is concerned
Satisfied	a green tick	your work is compliant on this criterion
Unsatisfied	a cross, then "non satisfait" (unsatisfied) in the summary	your work needs fixing
Not evaluated	(ERROR) during the run, then a question mark and "non évalué : vérification impossible"	the setup, not you
Skipped	⊘ and (skipped), absent from the summary	the test assumes a condition that is absent

Swipe to see more

One last situation is neither a failure nor a success: nothing was measured. check says so, exits with code 1 and records no attempt, hence no mark. It is not a failed exercise, and the two cases that lead to it do not have the same cause.

If no test could be collected, pytest could not start: a conftest.py that raises, an unreachable machine or a missing dependency. The output just above the message says which one.

If all the tests were skipped, the lab is not ready: its working directory was not prepared, or a service it needs is not running. check says so and gives the command to run first, dsoxlab run <id>.

What does check show of a failed criterion, depending on the lab type?

The lab's type sets what check reveals of a failed criterion: everything on a guided lab, a short hint on a challenge, nothing but the verdict on a mock exam. This choice protects the value of the exercise. A guided lab teaches, so it can say everything. A challenge must not hand over the solution along with the statement of the failure. A mock exam measures, and gives nothing.

Lab type	What check shows of a failed criterion	With dsoxlab -v check
Guided lab (lab)	The author's full message	The full output of the tests
Challenge	The criterion's label, its verdict and the first line of the author's message	The full output of the tests
Mock exam (capstone, or any lab, challenge included, that declares a pass mark)	The criterion's label and its verdict, no message	Nothing more: the detail is not available

Swipe to see more

On a challenge, the first line says where to look and the following ones, which carry the repair gesture, are not shown. On a mock exam, even dsoxlab -v check does not display the raw output of the tests: only the verdict per criterion is displayed, followed by a line saying that the detail is not available. Here is a check played on the epreuve-livrer-une-image mock exam of the containers catalogue, having done nothing, after taking two hints:

$ dsoxlab check epreuve-livrer-une-image
ℹ Validation de epreuve-livrer-une-image…
  ✘ test_functional.py::test_la_livraison_est_publiee_et_designee_par_son_digest
(FAILED)
  ✘ test_functional.py::test_le_secret_a_servi_et_ne_survit_dans_aucune_couche
(FAILED)
  ✘ test_functional.py::test_le_client_nu_tire_par_digest_et_execute  (FAILED)
  ✘ test_functional.py::test_le_compte_en_lecture_seule_ne_peut_pas_publier
(FAILED)
  Tests epreuve-livrer-une-image terminés ━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:02
Ce qui reste à faire, d'après les tests :
  ✘ La livraison est publiée et un client nu la récupère et l'exécute (non
satisfait)
  ✘ Le secret de construction a servi sans laisser de trace dans l'image (non
satisfait)
  ✘ Un compte en lecture seule ne peut pas publier dans le registre (non
satisfait)
╭─ Résultat : epreuve-livrer-une-image ─╮
│  0 / 100 pts                          │
│  ░░░░░░░░░░░░░░░░░░░░  0/4 tests      │
│  2 indices utilisés, -25 pts          │
╰───────────────────────────────────────╯
ℹ Résultat enregistré dans l'historique (0/100 pts).
✘ Des tests ont échoué.

Look at the "Ce qui reste à faire" list: each criterion carries its label and "non satisfait" (unsatisfied), with no help message at all. It has three criteria although four tests ran: a criterion can group several tests, and the first label speaks both of publication and of the bare client pulling the image. The panel recalls the two hints taken, which take 25 points off the scale. The command exited with code 1.

With dsoxlab -v check, the output is the same, but a line is added after the list to say that the detail is not available:

$ dsoxlab -v check epreuve-livrer-une-image
[...]
  ✘ Un compte en lecture seule ne peut pas publier dans le registre (non
satisfait)
ℹ C'est une épreuve : le détail des tests n'est pas disponible, seul le verdict
par critère l'est.
[...]

The -v option is global: it goes before the command name. Typed after, it is refused:

$ dsoxlab check epreuve-livrer-une-image -v
Usage: dsoxlab check [OPTIONS] [lab_id]
Try 'dsoxlab check --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ No such option: -v                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

The exit code is 2, a usage error. The valid form remains dsoxlab -v check <lab>.

The machine mode, dsoxlab check --json, follows the same rule. Besides the score, it returns failed_checks and criteria, and on a mock exam its output field is empty and output_withheld is true. A script therefore cannot get around the lab's policy by reading the JSON document. Here is the document of a failed check --json on epreuve-livrer-une-image, abridged to the fields a script uses:

$ dsoxlab check epreuve-livrer-une-image --json
{
  "schema": 1,
  "lab": {
    "id": "epreuve-livrer-une-image",
    "type": "challenge",
    "exam_passing_score": 75,
    [...]
  },
  "check": {
    "ok": false,
    "passed": 0,
    "total": 4,
    "score": 0,
    "max_score": 100,
    "recorded": true,
    "status": "failed",
    "output": "",
    "output_withheld": true,
    "failed_checks": [
      "test_la_livraison_est_publiee_et_designee_par_son_digest",
      [...]
    ],
    "criteria": [
      {
        "id": "livraison-utilisable",
        "state": "unsatisfied",
        [...]
      },
      [...]
    ]
  }
}

Look at output and output_withheld: the raw output is empty and the flag says why. The verdict is read in criteria, where state is here unsatisfied, a stable token that does not depend on the language. The type field is challenge: that is how this mock exam declares itself in its lab.yaml, and it is its exam_passing_score that makes it a mock exam. The command exited with code 1.

The level of detail is the one the catalogue's author declares with the feedback: field of the lab.yaml (full, first-line or verdict). Only the author can reveal more on a mock exam, never the learner.

How is a mock exam judged?

A mock exam is a lab that declares a pass mark, and submit returns its verdict: passed or refused, expressed as a percentage of the scale. The CLI messages say either « épreuve » or « examen » (mock exam): « Examen réussi » (exam passed), « Examen échoué » (exam failed) and « Épreuve refusée » (mock exam refused) all three close a mock exam. A mock exam can also declare blocking checks, criteria whose failure is enough to refuse the exam whatever the total. Here is the end of a submit on epreuve-livrer-une-image, submitted having done nothing, with a pass mark of 75% (the beginning of the output repeats that of a check, so it is abridged):

$ dsoxlab submit epreuve-livrer-une-image
[...]
╭─ Résultat : epreuve-livrer-une-image ─╮
│  0 / 100 pts                          │
│  ░░░░░░░░░░░░░░░░░░░░  0/4 tests      │
│  2 indices utilisés, -25 pts          │
╰───────────────────────────────────────╯
ℹ Résultat enregistré dans l'historique (0/100 pts).
✘ Épreuve refusée pour deux raisons. Un contrôle bloquant a échoué, et aucune
note ne le compense :
  test_le_secret_a_servi_et_ne_survit_dans_aucune_couche
Et votre total, 0 % du barème, reste sous le seuil de 75 %.

Tentative enregistrée. Corrigez l'exigence ci-dessus, puis dsoxlab submit à
nouveau.

Look at the final verdict: it gives two reasons for the refusal, a failed blocking check, whose test it names, and a total under the pass mark. The test named is the one to fix first, because no mark makes up for a blocking check: with a mark above the pass mark, that single test would be enough to refuse the exam.

The two other verdicts are simpler. When everything passes, submit concludes with "Examen réussi : 100 % du barème, pour un seuil de 75 %" (Exam passed: 100% of the scale, for a pass mark of 75%). Without a blocking check, a mark under the pass mark gives "Examen échoué" (Exam failed) and invites you to play the lab again with dsoxlab reset, then dsoxlab run.

What does each recorded attempt keep?

Each pass of check or submit records an attempt, with enough to reread it afterwards. It carries a unique identifier, the lab's version at the time of the pass, the state of each criterion and the hints you had seen, with their identifier. These hints remain after a reset, which nevertheless erases the lab's hint requests: the history therefore always says what you had seen when you validated. dsoxlab scores relies on this memory.

Optional: what the local database keeps, and what leaves it

The export and dsoxlab push rely on this memory too. For each criterion, the local database keeps the identifier, the state and the message written by the lab's author, in full. The lab's policy only limits what leaves that database: the export only takes the first line of it, and nothing on a mock exam, and dsoxlab push sends your account only the identifier and the state of each criterion, on a catalogue that declared it. The technical text of an exception is never kept.

Two cases record nothing: a validation lab, which defends a published guide and grades nobody, and a run that measured nothing, as seen above.

How do you read the state of a lab with dsoxlab status?

dsoxlab status says where the active lab, or the one you name, stands, with a stable state: not_started, ready, in_progress, validated or degraded. It never speaks of the machines: that is the role of dsoxlab infra status, which verifies that each host answers on SSH. On a prepared vm lab, a panel titled "État du lab" (Lab state), followed by the lab's name, displays en cours (in progress) and the sentence "Préparé, le travail se fait sur la machine du lab." (Prepared, the work is done on the lab's machine.). With no active lab, status says so and points to run, which is what the answer shows after a submit, which empties the active lab:

$ dsoxlab status
ℹ Aucun lab actif dans ce catalogue. Choisis-en un : dsoxlab run <lab>
ℹ Pour l'état de l'infrastructure, c'est désormais : dsoxlab infra status

Look at the second line: for the state of the machines, type dsoxlab infra status, not status.

State	Label	What it means
not_started	non commencé (not started)	Nothing is prepared: dsoxlab run <lab>
ready	prêt (ready)	The environment is prepared, nothing has been touched in it
in_progress	en cours (in progress)	The work has begun: the working directory differs from what run put there, or the preparation of a vm lab took place
validated	validé (validated)	A mark has been recorded for this lab
degraded	dégradé (degraded)	A declared service no longer runs, or the Docker engine is missing or does not answer

Swipe to see more

Beware of the word "validated": it means that a mark exists, not that all the tests pass. A lab marked 0 out of 100 is "validated" in the sense of status, which then displays "Note obtenue : 0 / 100" (Mark obtained: 0 / 100). To know whether a mock exam was passed, look at the verdict in dsoxlab scores. The degraded state takes precedence over the others, because it calls for an immediate gesture: restart the service with dsoxlab run <lab>. dsoxlab status --json returns the state as a stable token, which you can filter without depending on the language.

Troubleshooting

None of these cases is a breakdown of the tool: each time, the message says what is missing or what is refused.

Symptom	Cause	Solution
check exits with 2 and asks you to provision	vm lab without infrastructure; nothing is recorded	dsoxlab provision, then run check again
check reports that no test could be collected, and records nothing	pytest could not start	Read the output: a conftest.py that raises, an unreachable machine, a missing dependency; dsoxlab doctor
check reports that all tests were skipped, and records nothing	The lab is not ready: working directory not prepared or service stopped	dsoxlab run <id>, then run check again
A criterion shows "non évalué : vérification impossible" (not evaluated: verification impossible)	The setup could not verify this criterion	Run check again, then dsoxlab doctor; it is not your fault
No help message under a failed criterion	The lab is a mock exam, or a challenge that only shows the first line	This is intended: reread the mission with dsoxlab challenge
dsoxlab -v check does not display the full output	The lab is a mock exam	No detail is available on a mock exam
dsoxlab check <lab> -v answers "No such option: -v"	-v is a global option, it goes before the command	dsoxlab -v check <lab>
submit refuses a mock exam at 80% for a pass mark of 75%	A blocking check failed	Fix the test named, then dsoxlab submit
status displays "validé" (validated) with 0 / 100	"validated" means a mark exists	Reread the summary of check, fix, run again

Swipe to see more

Key points

• check records an attempt at each pass, a failed one included; submit does the same then closes the session.
• A criterion is satisfied, unsatisfied, not evaluated or skipped; not evaluated is not your fault, and a skipped test does not appear in "Ce qui reste à faire".
• The feedback depends on the lab type: full message on a guided lab, label and first line on a challenge, verdict alone on a mock exam.
• On a mock exam, even dsoxlab -v check and check --json give no detail (output_withheld), and a failed blocking check refuses it whatever the total.
• Nothing is recorded if no test could be measured, nor for a validation lab.
• -v goes before the command: dsoxlab -v check <lab>.
• status says where the lab stands; "validated" means a mark exists.

Knowledge check

Check that you have the essentials of this guide. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

8 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →