Lesson 9 · Verification, extended

Metamorphic relations, and where the inputs come from

An extension of Lesson 4, which named axis 3 and left the empty cell in your test matrix as an observation. This is the mechanism for filling it — and a way to tell a strong property from a decorative one.

A metamorphic relation is a pair

The reason “metamorphic testing” feels vague until it suddenly doesn't is that people describe it by example (round-trip!) rather than by its definition. It is a pair:

T : Input  → Input      the input transformation
R : Output × Output → bool   the output relation

MR:  for all x.  R( f(x), f(T(x)) )

That's it. You pick a way to perturb the input, and a way the two outputs must relate. Round-trip is one instance where T is “print then reparse” and R is equality. Once T and R are separate objects in your head you can enumerate properties instead of waiting for them to occur to you — pick a T, then ask what survives.

You are not looking for properties. You are looking for pairs.

Two of those pairs have names worth having, because they are the two that catch the most:

invariance	`f(T(x)) == f(x)` — the output doesn't move at all. Only correct when your tool is deliberately blind to what `T` changed.
equivariance a.k.a. naturality	`f(T(x)) == T'(f(x))` — the output moves the same way the input did. This is the workhorse, and the one people miss: they try invariance, find it false, and conclude there is no property there.

Getting T' right is the whole craft. T renames a field in the source; T' renames it in the expected output. If you can write T', you have a property that needs no expected file — for any input, including generated ones.

Exercise A — Grade the pairs

Sixteen combinations. For each, decide what kind of property it is. There are five verdicts, and each is used at least once. Explore the grid — the tally below tracks what you've graded.

Verdicts. holds · strong worth writing and it can fail · holds · weak true, but implied by a stronger pair here · false there is a counterexample · needs a precondition true only on a restricted T · vacuous cannot fail, so it tests nothing.

Where the inputs come from

Every property above quantifies over inputs, and your inputs are hand-written module literals. To generate them you have three options, and two of them are traps.

Generate IR st.builds(Model, …)	Cheap, and it skips the front end — so it cannot fill the empty cell in Lesson 4's matrix, because that cell is real source through the loader, then passes. Useful for testing pass algebra in isolation, which is what `test_transformers.py` already does by hand.
Generate source text	The trap. You would be writing a Python emitter in order to test a Python emitter, and every bug in yours arrives disguised as a bug in the tool.
Generate a spec	The answer. Generate a small description, then derive two things from it: the input module's source, and what you expect to be true of the output. The description is simpler than either — that asymmetry is what makes it an oracle rather than a mirror.

That third option has a name: model-based testing. The spec is the model, deliberately dumber than the implementation.

class FieldSpec(BaseModel):
    name: FieldName
    annotation: AnnotationText

names = st.from_regex(r"[a-z][a-z_]{0,7}", fullmatch=True).map(FieldName)
annotations = st.sampled_from(["int", "str", "UUID", "list[int]", '"Later"']).map(AnnotationText)

field_specs = st.builds(FieldSpec, name=names, annotation=annotations)
model_specs = st.builds(
    ModelSpec,
    name=names.map(lambda n: ModelName(n.root.title())),
    fields=st.lists(field_specs, min_size=1, max_size=6,
                    unique_by=lambda spec: spec.name.root).map(tuple),
)

pipelines = st.lists(st.sampled_from([partial_none(), partial_sentinel(), omit_first()]), max_size=3)

Then write the spec to a temporary module, import it, and run the real load → pipe → generated path. You already have the import machinery — executed(source, ModuleName(...)) in the corpus tests does exactly this for outputs; point it at inputs and the empty cell is closed.

Four rules that decide whether this works

Build, don't filter	`unique_by=`, not `.filter(lambda fs: no_duplicates(fs))`. Every filtered candidate is thrown-away work, and filters make shrinking worse because the shrinker keeps proposing smaller cases the filter rejects. `assume()` is the same trade, and both are fine in small doses and fatal as a habit.
No `random`	All entropy comes from the strategy. One `random.choice` inside a generator and you lose both reproducibility and shrinking, which are the two things you came for.
Cap the recursion	Models referring to models needs `st.deferred` and a size budget — the same divergence problem as `fake()` in Lesson 5, with the same fix from the same QuickCheck paper.
Generate the pipeline	The input is not just a model; it's (model, recipe). Generating the pass list turns Lesson 4's hand-written commutativity table into a property, and when the recipe is a sequence of operations the technique has a name: stateful or model-based testing — Hypothesis spells it `RuleBasedStateMachine`.

The property to write first

Before any of the equivariance work, there is one property that needs no T', no expected output, and no spec — and for a tool whose entire discipline is a declared supported fragment, it is the highest-value assertion available:

@given(module_specs(include_unsupported=True), pipelines())
def test_total(spec, pipeline):
    try:
        write(recipe(spec, pipeline))
    except UnrepresentableError:
        pass                      # a refusal is a correct outcome

Every other exception type is a bug: an AttributeError out of the loader, an IndexError from a regex that found nothing, a KeyError in the bindings table. This is a robustness or totality property — the tool is a total function into “file or refusal” — and it is exactly the guarantee rejections.py claims. Nobody has ever checked it against an input nobody wrote by hand.

It also composes with Lesson 6: once diagnostics are values, the same property gets sharper — a run either writes every file or reports at least one diagnostic, and never both.

Keeping a property honest

The failure mode of property-based testing is not a false alarm. It is a green suite of properties that cannot fail.

Check the distribution, not just the pass

Your generator may never produce the interesting case. If annotations never emits an already-optional type, every property about partial_none passes without testing the branch that matters. Hypothesis gives you event() to label cases and --hypothesis-show-statistics to print the distribution; target() steers the search toward inputs you say are interesting. Look at the numbers once per property, at least.

Break the code on purpose

The only real evidence that a property has power is watching it go red. Delete the bracket-collapsing loop in _widened, invert a comparison in imports_of, make _own_fields return everything in the MRO — and check which properties notice. Doing this systematically is mutation testing, and it is how Hughes's How to Specify It! compares specification styles: one function, eight seeded bugs, and a count of which properties caught what. A property that survives every mutation is decoration.

Pin what you find

A failing property shrinks to a minimal case. Do not leave that case in the search — copy it into an @example(...), or better, into test_corpus.py as a named module, so it is checked deterministically forever. Random search finds a bug once; a corpus entry finds the regression every time.

Repo-specific hazard. Two layers of your test setup exist to avoid rerunning tests, and both are wrong about properties. :test runs under --testmon, which reruns tests whose dependencies changed — but a property's input space isn't a file, so nothing looks changed; and moon caches the task on declared inputs, so a clean cache replays a pass. A property test that only ever runs when you edit the file it lives in is a property test that doesn't run. Give properties their own uncached task with a fixed seed budget, keep the .hypothesis/examples database out of the cache key, and let the corpus stay fast and cached.

Exercise B — Recall

The counterweight

A metamorphic relation constrains a family of outputs; it never pins down which member of the family you emit. Consider a generator that emits class X: pass for every input: it is equivariant under renaming, invariant under permutation, idempotent, a fixed point, and total. It passes almost the whole grid above.

Properties say the output is consistent. One golden file says it is right. You need both, and they are not substitutes.

So the shape of a finished harness, and it is only a small step from what you have: a small corpus of real modules with golden output (intent, and fast), execution of every generated file (cheap totality), generated specs driving the equivariance pairs across the transformer matrix (the empty cell), and the totality property over deliberately-unsupported input (the fragment boundary, kept honest). Four things, and Lesson 4's four axes map onto them one for one — with the inputs no longer chosen by whoever wrote the test.

status · resources · mission