Lesson 4 · Verification

Testing a code generator

You asked to learn this. You are already doing three of the four things — this names them, and finds the gap they leave.

Testing a generator has one unusual asset and one unusual difficulty.

The asset: the output is a program, so you can run it. Most functions produce data you have to inspect; yours produces something that can be asked to justify itself.

The difficulty is the oracle problem — the general name for “how do you know the right answer?”. For a generator the right answer is a whole file of text, and nobody wants to write those by hand. Every technique below is a different way of dodging the need to know the exact right answer.

The four axes

1 · Golden tests (also: snapshot, characterisation, approval)

assert result.body == PythonSource("""\
class Note(BaseModel):
    body: Annotated[str, StringConstraints(min_length=1)]
""")

Oracle: recorded. You ran it once, eyeballed the output, and froze it.
Catches: any unintended change to output, exactly and immediately. Superb regression net.
Misses: whether the frozen output was ever correct. A golden test records the bug alongside the behaviour, with equal confidence. Michael Feathers' term characterisation test is the honest one — it characterises what the code does, not what it should do.

2 · Execute the output

with executed(source, ModuleName(f"generated_{bound.root}")) as module:
    generated_model = getattr(module, bound.root)

Oracle: the Python interpreter.
Catches: syntax errors, missing imports, unbound names, import-time explosions — the entire class of bug where you emitted something that isn't a program. It is the cheapest high-value test a generator can have, and most generators don't have it.
Misses: anything that runs but means the wrong thing.

3 · Metamorphic properties (round-trip is one)

assert result.generated == result.source   # comparing _shape(...)

Oracle: derived. You don't state the answer — you state a relation between input and output that must hold whatever the answer is. This one says: with no transformers applied, the generated model has the same field shape as the model it came from.
Catches: semantic drift, at any input, without anyone writing an expected file.
Misses: exactly and only what the relation doesn't look at. Yours compares annotation | metadata | is_required. It is blind to defaults, and blind to how a default is produced — which is the tier-3 loss from Lesson 2, invisible to this test by construction.

This is the technique with the best power-to-effort ratio and the one most people never reach for, because it requires a shift: from “what should the answer be?” to “what must be true of the answer?”. Round-trip (parse(print(x)) == x) is the famous instance, but the family is much wider — idempotence, commutativity, monotonicity, invariance under renaming.

4 · Negative tests on the fragment boundary

test_corpus_unrepresentable.py: assert that unsupported input is refused, by the specific error.
Catches: silent acceptance — the failure where you quietly emit something plausible for input you don't actually support. For a tool whose whole discipline is a declared supported fragment, this is the axis that keeps the declaration honest.
Misses: input you never thought to reject.

Exercise A — Name the axis

Four assertions. Which axis, and what is each one blind to?

The axis you don't have: generated inputs

All four axes above answer “is the output right?”. None of them answers “did we try the input that breaks it?” — and every input in your corpus is a hand-written string literal. You test the inputs you thought of.

Property-based testing closes that: generate the inputs, and check the metamorphic properties over all of them. Hypothesis is the Python one; it also shrinks a failing case to the smallest input that still fails, which for a generator is the difference between a 40-line repro and a 2-line one.

Here is why this matters concretely, and it's the sharpest thing in this lesson. Look at how your suite is partitioned:

The bottom-left cell is empty. Real source goes through the loader with no passes; passes are exercised on AnnotationText("Name") literals that never came from a file and are never rendered, executed, or round-tripped.

Your corpus contains a module with note: "Tag | None" = None. Your transformer tests contain partial_none(). The bug from Lesson 3 lives precisely in the cell where those two never meet. It isn't a gap in rigour — each half is tested carefully. It's a gap in the cross product, and cross products are exactly what hand-written cases don't cover and generated ones do.

Hand-written tests cover the cases you imagined. Generated tests cover the combinations you didn't.

Extension: Lesson 9 takes this on properly — metamorphic relations as (transformation, relation) pairs, sixteen of them graded, and how to generate inputs that actually reach the loader rather than the IR.

Pass algebra — your question 4, properly

You said idempotence isn't enough, which is right, and is the interesting half of knowing it. Three properties, and they are not the same:

So the answer: a set of passes needs to be pairwise commutative for order never to matter, and confluent for that to survive as the set grows.

And the second half of your question — do yours have it? No. Two counterexamples, both reachable from the public recipe API:

pipe(m, omit("a"), pick("a"))   →  UnknownFieldError: … has no field a
pipe(m, pick("a"), omit("a"))   →  class M: pass

pipe(m, partial_none(), partial_sentinel())
    →  x: int | None | MISSING = MISSING
pipe(m, partial_sentinel(), partial_none())
    →  x: int | MISSING | None = None

The second pair is the worrying one: no error, two different files, and the default differs — so the two orders produce payload models with genuinely different behaviour.

The move is not to force commutativity. Most real compilers aren't confluent either; LLVM's pass ordering is a known, documented, permanently-annoying fact of life. The move is to know which pairs commute, assert the ones that do, and document the ones that don't — because right now the recipe author has no way to find out except by trying it.

Exercise B — Do they commute?

For each pair, does swapping the order change the result?

Your question 10, cashed out

You said: round-trip and inverses, worth a test. Correct — and you've already built it (_round_trip, _shape). Three properties are sitting one line away from the harness you have:

Note what all three have in common: none requires anyone to write down an expected output. That's the whole trick of metamorphic testing, and it's why it scales to a generator where golden files don't.

Exercise C — Which axis catches it?

Four bugs. Pick the cheapest axis that would catch each. One has an uncomfortable answer.

status · resources · mission