Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

The work loop with a code agent: specify, test, review the diff

By updated

Improve this page

A code agent produces what you ask it to make true, and it takes the shortest path to get there. If the only instruction is "make the tests pass", deleting a failing test is a valid path. This lesson gives the loop that prevents it: write a verifiable specification, split it into small slices, have the tests written before the code, then refuse any diff that deletes or weakens a test. It is written for developers and for people in a business role who produce code with AI. It does not depend on any particular assistant: a code agent such as Claude Code (in French), an editor assistant or an online generator will do, as long as you can read the diff before accepting it.

What you will learn

  • Write a requirement with an acceptance criterion tied to a named test.
  • Split a request into slices you can review in a few minutes.
  • Have the test written before the code and review it before starting the generation.
  • Spot a diff that deletes, skips or weakens a test.
  • Refuse that diff and record the reason so its author can find it again.

Prerequisites

You can read a git diff and make a commit. The lesson Saving changes: add, diff, commit teaches git add, git commit and git diff, and the lesson Browsing the history: log, show, HEAD teaches git log and git show. You have already seen a test written with pytest (in French): the examples are in Python, but the reasoning applies to any language. You can create a virtual environment and install a package in it with pip, which the page Python virtual environments: venv, pip and requirements.txt (in French) explains. You can read and write a small Python program, as the lesson First steps in Python: program, variables and types (in French) shows, and you can run a command in a Linux terminal, which the lesson Open a terminal and understand the prompt teaches. The "Technical foundations" module of the course brings these lessons together; it is optional for anyone who already knows how to do all this. To type the commands of the lesson, you also need Python 3.11 or later and pytest installed.

Why is a green test suite not enough?

A green test suite says that nothing failed, not that every test ran. An agent that has to "make the tests pass" has three ways to do it: fix the code, change the test so that it accepts the current result, or remove the test. The last two also give a green suite, and they are faster than the first.

Vibe coding in the strict sense does not look at the diff, so it sees none of this. AI-assisted development does look at it, but the eye must know what to look for. Take the purchasing portal, the running example of the course, from the side of the buyer who generated it. She asks the agent: "a blocked supplier must no longer be able to receive an order". The agent changes the code, an existing test fails, and the diff it proposes also contains a @pytest.mark.skip line added on that test. The suite passes. The rule is still violated, and nobody knows it any more.

The loop below does not rely on trust in the agent. It puts the decision at a place the agent does not control: the specification and the tests, written and reviewed before the generation.

Work loop in four steps: a specification with criteria, tests written first and red, the generation of a slice by the agent, then two checks. If the tests fail, the agent fixes the code. If they pass, we ask whether the diff touches the tests: if a test is deleted or weakened, the diff is refused and the reason recorded; otherwise the slice is accepted and we move to the next one.noyesyes, test removedor weakenednoSpecificationwith criteriaTests writtenfirst, redThe agent generatesone sliceDo the testspass?Does the diff touchthe tests?Refuse the diff,log the reasonAccept the sliceNext slice
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Look at the two diamonds: that is where the method does its work. The first one ("do the tests pass?") is the one everybody looks at. The second one ("does the diff touch the tests?") is the one that gets forgotten, and it is the one that stops the agent that cheats. The test written first, at the start of the line, is what makes the second diamond possible: you can compare what the agent proposes with what you had decided. The section "How do you spot a diff that removes or weakens a test?" gives three checks to answer that question.

How do you write a verifiable specification?

A specification is verifiable when each requirement has an acceptance criterion tied to one precise test. Without a criterion, nobody can say whether the code meets the requirement, neither the agent nor you. The format has three elements: the requirement in one sentence, the criterion that says how it is observed, and the identifier of the test that proves it.

Here is a requirement of the purchasing portal, written in a SPEC.md file at the root of the project. The test identifier is the one pytest prints, in the form path::name.

- **R4** : an order addressed to a supplier with status `bloque` is
refused. An order addressed to a supplier with status `actif` is accepted.
- AC-4 [R4] : creating an order for a `bloque` supplier raises an
error `FournisseurBloque`.
Test : `tests/test_commandes.py::test_fournisseur_bloque_refuse`
- AC-5 [R4] : creating an order for an `actif` supplier succeeds.
Test : `tests/test_commandes.py::test_fournisseur_actif_accepte`

The requirement describes the behaviour, not how to code it. The two criteria cover the case that must fail and the one that must succeed: a test that only checks the refusal would let through an implementation that refuses everything. The content of the criterion also stays as close as possible to what can be observed ("raises an error", "totals 224.80"), not to an adjective ("handles correctly"). An adjective cannot be tested.

This specification has another use: it is the first text you give to the agent. The guide basic prompting with Claude Code (in French) explains how to phrase a request; the specification is its core, and it stays valid if you change assistant.

How do you split the work into slices?

A slice is the smallest change that makes a requirement true, and whose diff can be read in a few minutes. The practical rule: one requirement, one or two tests, one diff. Asking "add all the supplier management" produces a diff of several hundred lines that nobody really reads, and the risk comes back through the door above: you accept because it is too long to read.

For the purchasing portal, the request "block suppliers" splits as follows, each line being a slice that you accept or refuse on its own:

  1. R4, a blocked supplier receives no order;
  2. R5, a supplier can be blocked and unblocked by a manager;
  3. R6, the block is visible in the list of suppliers.

A slice ends with a commit. If the agent did two things in the same diff, refuse the whole diff and ask for the slice alone again: a mixed diff hides what it really changes. The guide building a complete project driven by Claude Code (in French) takes up this split as "vertical slices" on a whole project.

Why have the tests written before the code?

When the test exists before the code, it expresses what you decided, and the agent can no longer adapt it to what it produced. If the agent writes test and code in the same motion, the test describes what the code does, not what it should do: it always passes, and it protects nothing.

The gesture has four steps for each slice:

  1. Ask the agent to write only the test of the criterion, without touching the code.
  2. Review that test. Check that it really verifies what the criterion says: the error case, the expected value, not only the absence of an exception.
  3. Run it: it must fail, and fail for the right reason. With pytest, pytest tests/test_commandes.py::test_fournisseur_bloque_refuse must show a failing test, because the behaviour does not exist yet. A test that passes before the code exists tests nothing.
  4. Then ask the agent to write the code that makes it pass, without modifying the test.

The test written by the agent is therefore not the problem, the test nobody reviewed is. The guide testing strategies explains why writing the test first forces you to specify what the code must do, and the page advanced tests in Python (in French) shows how to write tests that really detect defects.

How do you spot a diff that removes or weakens a test?

Three checks let you recognise a diff that touches the tests. Run them before accepting the slice. Before starting the agent, note two reference points: the number of tests collected, with pytest --collect-only -q | tail -1, and the starting commit. Without them, you cannot say afterwards whether a test has disappeared. In this command, the | sends the output of pytest to tail -1, which keeps only the last line, the one that gives the total.

The commands below compare the working folder with the last commit using git diff HEAD, which covers staged changes (git add) as well as the others. git diff without an argument only shows changes not yet staged: it sees nothing of what the agent staged. If the agent committed the slice, git diff HEAD shows nothing any more: replace HEAD with HEAD~1 (or with the starting commit if the agent made several commits), or read git show --stat. HEAD designates the last commit, and HEAD~1 the commit just before it: once the slice is committed, comparing with HEAD~1 shows what the agent's commit changed.

The first check is to look at which files the diff modifies.

Terminal window
git diff HEAD --stat

The output lists the modified files and the number of lines. Look in it for any path under tests/, but also conftest.py, pytest.ini, pyproject.toml or the continuous integration configuration file: an option such as --deselect, a reduced testpaths or an added -k remove tests from the suite without deleting a single one. If the slice concerns the code of a requirement, none of these files should appear. If one does, read it first.

The second check is to look for the removed lines and the words that neutralise a test in the part of the diff that concerns the tests.

Terminal window
git diff HEAD -U0 -- tests/ | grep -E '^[-+].*(skip|xfail|assert|raises)'

The | sends the output of git diff to grep. The pattern '^[-+].*(skip|xfail|assert|raises)' keeps the lines added (+) or removed (-) that contain one of these four words.

A - line carrying an assert is a deleted verification. A + line carrying skip or xfail disables a test. A - line that held a precise value, replaced by a looser assert (assert total == Decimal("224.80") turned into assert total), is a weakening: the test passes for almost any result. This pattern does not see a test renamed so that it is no longer collected, nor a change that contains none of these words: the third check spots the renamed or skipped test.

The third check is to count the tests that ran, not just read the word "passed". The -rs and -rx options ask pytest to add, at the end of the run, a "short test summary info" section that lists the skipped tests and the tests expected to fail. The exact wording of the lines varies from one pytest version to another: the outputs below give the form, taken with pytest 9.2.0 and Git 2.47.3, to be checked on your version.

Terminal window
pytest -rsx

For a test skipped by @pytest.mark.skip(reason="flaky"), the section contains a line of the form SKIPPED [1] tests/test_commandes.py:14: flaky. It gives the file, the line and the reason, and the number in brackets counts the tests skipped for that same location and that same reason. An s also appears in the line of dots. The suite is not red, but a test did not run.

Then compare the number of tests collected with the one noted before the agent, using pytest --collect-only -q | tail -1 (the same command as above: the | passes the output to tail -1, which keeps the last line). The line has the form 2 tests collected followed by the duration. When -k or --deselect sets tests aside, it becomes of the form 1/2 tests collected (1 deselected). If the number of tests kept goes down while you removed nothing, a test has disappeared.

This third check does not see a weakened test: it is still collected and it passes, so neither -rsx nor the number of tests changes. Only reading the diff of tests/, the second check, finds it: a looser assert shows in the - and + lines. When a slice touches a file of tests/, read that diff in full.

A modified test is not always a weakened test: a requirement that changes justifies a change of test. The rule is that this modification comes from you, before the generation, never from the agent while it generates. To review a diff with method, the page reviewing an automation (in French) teaches how to do it without reading every line, and the guide plan mode, diff and validations (in French) shows how to force the agent to propose a plan and wait for your agreement before writing.

What should you do when you refuse a diff?

Refusing a diff takes two steps: do not apply it, and write down the reason so that it serves the agent, your team and the next review. A useful reason says precisely what is refused: the file, the test, the line.

- tranche: R4, fournisseur bloqué
decision: refusee
raison: >
Le diff ajoute @pytest.mark.skip sur
tests/test_commandes.py::test_fournisseur_bloque_refuse
(ligne 14). Le critère AC-4 n'est plus vérifié. Redemander la tranche
sans toucher aux tests.

A reason such as "bad diff" is useless: the agent and you will find it in three weeks without being able to replay it. The refusal is then rephrased to the agent with the same precision: "do not modify tests/, fix commandes.py so that test_fournisseur_bloque_refuse passes". If the agent persists, it is a signal: stop the session and reframe, as the guide common mistakes and reframing (in French) explains.

What are the limits of this method?

This loop reduces one precise risk, code that does not do what you decided, but it does not remove it, and it leaves other risks open.

A test is worth only what the specification it translates is worth. If the requirement forgets a case, for example a supplier with an unknown status, the suite stays green and the defect gets through. Tests do not detect a security flaw that nobody described either: a missing access control makes no test fail that you did not write. This loop describes what you do. Detecting these defects automatically, with a code scanner for example, is another subject.

Finally, the method rests on your discipline: nothing technically prevents the agent from modifying a test if you do not forbid it. Writing the rule in the project instructions helps, enforcing it with an automatic check is safer, and that is another subject. Meanwhile, the three checks above let you hold the line by hand.

Troubleshooting

The table starts from what you observe after a slice and leads to the check to run.

SymptomLikely causeFix
The test written first passes straight awayIt does not verify the expected behaviour, or the code already existedRe-read the assertion; add the error case; make the test fail again before generating
The diff is several hundred lines longThe slice is too bigRefuse and ask again for one requirement at a time
The suite is green but pytest -rsx lists skipped testsA test received skip or xfailRefuse the diff, record the line, ask for the code to be fixed
The number of tests collected went downA test, a file or a configuration option was removedgit diff HEAD --stat (HEAD~1 if the agent committed) then read conftest.py and pytest.ini
The number of tests did not change, but a test passes for too many resultsAn assert was weakened, in a staged or committed changegit diff HEAD -U0 -- tests/ (HEAD~1 if committed) and read the - and + lines
The agent rewrites the test at every attemptThe instructions do not forbid itSay explicitly not to modify tests/; otherwise stop the session

Put it into practice

The lab takes up the purchasing portal, the running example of the course, and has you practise the gestures of this lesson on an orders module. It is recommended: it is the proof of the gesture. You need a terminal, Python 3.11 or later, pytest installed, and patch or git to apply a change.

A simulated agent proposes two changes to the orders module. You run the suite before touching anything, read each patch, apply the one that respects the specification and refuse the other one by recording the reason in the review log. You then write an acceptance criterion tied to a test for the new requirement. The validation reruns the whole suite: the accepted change must pass, every original test must still run and succeed, the totals must stay exact, the refusal must appear in the log with its reason, and the test of the criterion must fail as soon as the requirement is broken.

The files of the lab carry English names (orders.py, SPEC.md), not those of the purchasing portal in the lesson (commandes.py): the method is the same.

Key takeaways

  • A code agent optimises what you measure: a green suite can also be obtained by deleting, skipping or weakening a test.
  • Each requirement carries an acceptance criterion tied to a named test.
  • A slice is the smallest change that makes a requirement true, and its diff can be read in a few minutes.
  • The test is written and reviewed before the code, and it must fail for the right reason before the generation.
  • Before starting the agent, note the number of tests collected. Before accepting, check the modified files (git diff HEAD --stat, with HEAD~1 if the agent committed), the lines removed in tests/ and the number of tests that really ran. A weakened test only shows in the diff of tests/.
  • Refusing a diff is recorded with the file, the test and the line concerned.
  • The method replaces neither a complete specification nor a security review: a forgotten case remains an untested case.

Knowledge check

These eight questions cover the verifiable specification, the split into slices, the test written first and the refusal of a diff that weakens a test.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer