Lesson 11 · Diagnostics, part 3

Spans and provenance

Lesson 6 made errors into values, lesson 10 made recovery explicit, and both left every diagnostic with nowhere to point. Fixing that is not “add a line number”: in a code generator, half your nodes have no line number, and that is the interesting half.

Four words

Span	A region of one source file: a file identifier plus a start and end position. Not a line number — a range, because a diagnostic wants to underline the offending text, not gesture at the row it lives on. rustc interns these into a `Span` so small it fits in a register; you will not need that.
Provenance	Why a node exists at all. For a node you read from text, provenance is a span. For a node you invented, it is a derivation: this came from that, by that pass. A generator has both kinds, which is the whole design problem.
Primary / secondary	A diagnostic has one span it is about and any number of spans that explain it, each with a label. “Collision here (primary); the other import is here (secondary).” Most bad error messages are diagnostics that had two locations and were allowed to print one.
Source map	The reverse direction: output position → input position, so a stack trace in generated code can name the line you wrote. The source map spec is the JavaScript one. Note the direction carefully; it is the opposite of everything else in this lesson.

Only the first is standard equipment. The second is the one you actually need, and the reason is structural: pydantic-codegen's output contains nodes nobody wrote.

For a generator, provenance is a sum type

Three genuinely different kinds of node reach your renderer, and the reflex to give them all a span: Span | None is the reflex to throw the distinction away:

Read from the model	`Invoice.total` — a real `AnnAssign` in a real file at a real offset.
Read from the recipe	“generate a partial of `Invoice`” — also real text, but in a different file, and the thing it points at is a request rather than a declaration.
Synthesized	The `*overrides` parameter from lesson 7, the `MISSING` default from lesson 5. No location exists. It has a cause*, and the cause is a pass plus an input.

type Origin = ModelSpan | RecipeSpan | Derived

@dataclass(frozen=True)
class ModelSpan:
    module: ModulePath
    line: int        # 1-based
    col: int         # 0-based, UTF-8 bytes
    end_line: int
    end_col: int

@dataclass(frozen=True)
class Derived:
    by: PassName
    of: Origin

Derived is recursive, and that is the payload of the whole lesson. What a synthesized node has instead of a location is a chain, and a chain renders perfectly well:

error[E014]: mutable default shared across instances
  --> in code generated by `fake_constructor`
      from `partial_sentinel`
      from app/models.py:14:4  (Invoice.tags)

That is strictly more useful than a span would have been, because the user's fix is not at models.py:14 — it is a decision about the recipe. rustc reaches for the same shape and calls the pieces expansion context and macro backtrace: the “in this expansion of…” lines under a macro error are exactly a rendered Derived chain.

Three rules, and what enforces each

1. Allocate at the boundary, once

Only the loader may construct a ModelSpan or a RecipeSpan. A span is a claim about text that was read; a pass has read no text, so any span it constructs is a fabrication, and a fabricated location is worse than none — it sends the user to a line that is fine. Downstream code has exactly two moves: carry an Origin unchanged, or wrap it in Derived.

Same shape as Reported in lesson 10: a value that only one module can mint, so that possession of it proves something. Keep the two positional constructors private to loader.py and export only Derived.

2. On the node, not in a side table

The tempting cheap version is SPANS: dict[int, Span] keyed by id(node). It works until the first pass, because your IR nodes are frozen and every pass rebuilds them with replace() — a new object, a new id, and worse, an id that may be reused by CPython after the old node is collected, so the lookup returns a stale span rather than nothing. Provenance is a property of the node. Put it in the node.

3. A rewrite inherits; a construction must declare

These are the two ways a node comes out of a pass, and they have opposite defaults. replace(field, annotation=new) keeps origin for free — good. Field(name=…, annotation=…) does not, and if origin has a default it will silently be None forever. So: no default on the field. Every construction site is then forced to answer “where did this come from?”, and the ones that cannot answer are exactly the ones that should be writing Derived.

A required constructor argument is the cheapest static enforcement mechanism you own. Spend it on the fields that rot silently.

The Python facts you need

Five, and four of them are off-by-one hazards. This is the part that is nobody's theory and everybody's afternoon.

Mixed bases	`ast` nodes carry `lineno`, `col_offset`, `end_lineno`, `end_col_offset`. Lines are 1-based, columns are 0-based. Wrong choice of the two shows up as a caret one line down and one column left, which is exactly wrong enough to be blamed on the user.
Columns are bytes	`col_offset` is a UTF-8 byte offset, not a character index. Any non-ASCII earlier on the line — a name, a string literal, an em dash in a description — and `" " * col` overshoots. Convert through `line.encode()[:col].decode()` before you count characters.
`getsource` is relative	`ast.parse(inspect.getsource(cls))` gives you a tree whose `lineno` 1 is the `class` statement, not the file. `inspect.getsourcelines(cls)` returns `(lines, start)` — the offset you need — and `ast.increment_lineno(tree, start - 1)` applies it. Skipping this is the single most common span bug in Python tooling, and it fails quietly because the numbers look plausible.
Decorators	Since 3.8, `ClassDef.lineno` points at the `class` keyword, and the decorators are above it in `decorator_list`. “Where is this model?” therefore has two answers and you must pick one deliberately.
Inherited fields	A field declared on a base class lives in another module's text. So the module is part of the span and cannot be a global “the file we are working on” — the thing your loader already knows as `_declaring_class`.

The reference point worth reading is PEP 657, which added end positions to CPython tracebacks so a caret can underline a subexpression rather than a line. It is short, and it is honest about the cost — position tables are memory, which is why they were not there from the start.

Exercise A — Bug hunt

Five snippets. Click the line that carries the defect, or say no bug. Not all of them are broken; a wrong click reveals the answer, so commit before you click.

How much renderer to build

There are two levels and a large gap between them:

app/models.py:14:4: error[E014]: mutable default shared across instances

error[E014]: mutable default shared across instances
  --> app/models.py:14:4
   |
14 |     tags: list[str] = []
   |                       ^^ every fake() call would share this list
   |
help: emit a default_factory instead

The first is one line of code once you have an Origin, and it is clickable in every editor and every CI log — which is most of the value. The second needs the source text kept around, line slicing, tab handling, and the byte-offset conversion above. Build the first now. If you want the second later, the prior art is codespan-reporting, and Czaplicki is the standard for judging whether it was worth it.

The thing to do now, before either, is make the diagnostic type able to hold more than one location. Your best existing error is intrinsically two-placed: ShadowedImportError is about an import and a generated class, and a type with a single origin field guarantees you will print the less useful one. So:

@dataclass(frozen=True)
class Label:
    origin: Origin
    text: str

@dataclass(frozen=True)
class Diagnostic:
    code: DiagnosticCode
    message: str
    primary: Label
    secondary: tuple[Label, ...] = ()

The connection to hygiene

Worth knowing because it looks like a coincidence and is not. In Rust's procedural macros, a Span carries the hygiene context as well as the location, so Span::call_site() and Span::mixed_site() decide two things at once: where an error points, and what an identifier resolves to. Both questions have the same answer — “where was this written?” — and lesson 8 already told you that an identifier only means something together with its origin.

You should keep them as separate fields: bindings.py answers resolution, Origin answers location. But notice that they are computed from the same fact at the same moment, in the loader, and that a node which has one and not the other is a node whose loader took a shortcut.

Exercise B — Free recall

Interleaved with lessons 6 and 10, because these terms are easy to collapse into each other. Type the term; the field checks itself.

Exercise C — Decisions

Answer from memory, then reveal and compare.

What changes in the repo

`origins.py`	New. `Origin`, `ModelSpan`, `RecipeSpan`, `Derived`, `Label`. The two span constructors are private to the module; `loader.py` gets the only factory that can call them.
`ir.py`	`Model`, `Field` and `Import` gain `origin: Origin` with no default. Every construction site in `transformers.py` stops compiling, which is the point — each one is a decision about whether it is rewriting or inventing.
`loader.py`	Applies `ast.increment_lineno` at the point it parses, so nothing downstream ever sees a relative line number. Records the declaring module per field, not per model.
`diagnostics.py`	`Diagnostic` gains `primary` and `secondary`. `Poisoned(name, reported)` becomes `Poisoned(origin, reported)` — the `ModelName` was standing in for a location and can now stop.

And the test that keeps it from rotting, which is a property in the sense of lesson 4:

def test_every_diagnostic_points_into_a_real_file() -> None:
    # for each seeded defect: the primary label resolves, through any Derived chain,
    # to a (module, line) that exists in the input. Fails on fabricated spans and on
    # relative line numbers that were never incremented.

Note what that test is doing: Derived chains always bottom out in a real span, so “resolve to the root” is total, and a run whose chain bottoms out in nothing is a loader bug. That totality is why Origin is a sum with no None case.

The debt this creates

Origins make one thing suddenly visible. Lesson 3 mentioned in passing that Optional[X], Union[X, None] and X | None should be normalised to one form on the way in. Now ask what the diagnostic says afterwards. The user wrote Optional[int]; your canonical node says int | None; and if the message reports the canonical form, you are telling someone about code they did not write. A normalising pass erases exactly the distinctions a good error message needs — so the surface form has to survive somewhere, and the somewhere is the origin you just added.

That is the next lesson, and it also settles the half of lesson 4 you were one term short of.

status · resources · mission