Lesson 3 · IR design

How opaque should an IR node be?

Your question was “have I gone too low-level?” The answer is no — and the interesting part is where you went too high.

First, the direct answer

You asked when to use the Python AST, and whether you should have a different IR. You already have a different IR: ir.py. And you use the AST correctly — only in the front end, as a reader, never as the thing passes operate on. That separation is the whole game and you got it right. Three reasons it's right, worth being able to state:

Keep the AST as transport, not as IR. The distinction is worth naming because it's the mistake most codegen tools make: they parse to an AST, transform the AST, and unparse it, and every pass ends up re-deriving the same facts about the tree.

Now the real question

Your IR has a field called annotation: AnnotationText — a str in a wrapper. So does default: DefaultText. These are opaque nodes: the IR carries them, but the IR does not understand them.

Opacity is not a flaw. It's the single most useful dial in IR design, and the rule for setting it is:

An IR node must be exactly as structured as the most demanding pass requires — and no more.

For omit, pick, rename_model, set_bases: the annotation is a payload to be carried from input to output unchanged. Opaque is perfect. A structured type tree would be pure cost.

Then you wrote partial_none(). It has to answer: is this annotation already optional? That is a question about the annotation's structure. The box is no longer opaque. And since the IR gives you no structure, you reached for this:

outermost = annotation.root
while True:
    collapsed = re.sub(r"\[[^\[\]]*\]|\([^()]*\)", "", outermost)
    if collapsed == outermost:
        break
    outermost = collapsed
if any(term.strip() == token.root for term in outermost.split("|")):
    return annotation

That is a parser. A small, hand-rolled, undocumented parser for the Python type grammar, written in regex, inside a transform pass — for a language you already have ast.parse for.

A regular expression over IR text is the smell of a missing IR node.

Exercise A — Break it

Below is that exact function, ported to run here. Your job: find an annotation your loader accepts, for which partial_none() emits Python that raises at runtime.

It exists, and it's reachable — module_source._forward_refs goes to real trouble to support the construct in question, so it is squarely inside your supported fragment. The four existing tests in test_transformers.py don't cover it.

What the fix actually is

The tempting fix is a better regex. Don't. The bug isn't the regex — the bug is that a pass needs to see structure and the IR is hiding it. Every future pass that touches types (partial_sentinel already, and every projection, sum-type and Django-field pass you have planned) will need the same structure and will grow its own regex.

The fix is to make the node transparent, with an escape hatch:

class TypeName(BaseModel):      # int, User, Foo
    name: SymbolName

class TypeApply(BaseModel):     # list[int], Annotated[int, Field()]
    head: SymbolName
    args: tuple["TypeExpr", ...]

class TypeUnion(BaseModel):     # int | None
    members: tuple["TypeExpr", ...]

class TypeForward(BaseModel):   # "Foo"
    inner: "TypeExpr"

class TypeOpaque(BaseModel):    # everything else, verbatim
    source: AnnotationText

TypeExpr = TypeName | TypeApply | TypeUnion | TypeForward | TypeOpaque

Three things to notice, because they generalise past this example:

1. It is a sum type, and that is the point

An IR node is a sum of products: a closed set of alternatives, each with its own fields. In compiler writing this is so standard the languages are built for it — Rust's enum, ML's datatype, Haskell's data. Python got structural pattern matching in 3.10 specifically for this shape. A pass becomes a match that names every case, and adding a case makes the type checker point at every pass that must decide what to do.

2. TypeOpaque is not a cop-out — it is the design

The failure mode of “model the type grammar” is trying to be total over Python's types: Callable, ParamSpec, Literal, TypedDict, Annotated metadata, PEP 695 aliases. You'd be writing mypy. You don't need to. TypeOpaque lets you model exactly the constructs your passes interrogate and carry everything else verbatim, with source fidelity, forever. Partiality is the feature. The same instinct that produced rejections.py, applied one level down: you already know how to define a supported fragment and be loud about the edges.

3. You now get to delete the redundancy, not just avoid the crash

partial_none on Optional[int] currently emits Optional[int] | None. Valid, but it means your _widened only recognises one of the three spellings of optionality. With TypeExpr, Optional[X], Union[X, None] and X | None can be normalised to a single canonical form on the way in — a canonicalisation pass. Then partial_none is four lines with no string handling at all, and every later pass benefits from the same normalisation. That is the payoff of structure: facts get established once.

Exercise B — Set the dial

For each pass: does it need the annotation structured, or is opaque text enough? Answer before revealing.

The trap on the other side

Having just been told to add structure, the failure mode is to structure everything. Note what this lesson did not say: it did not say to structure DefaultText. No pass you have looks inside a default — they only replace it wholesale. Opaque is correct there, today.

This is the counterweight, and it's the harder half of the skill:

Structure is added in response to a pass that demands it, never in anticipation of one.

Speculative structure costs you constructors, tests, exhaustiveness burden in every match, and a migration when the real requirement turns out to be shaped differently. Wait for the second pass that needs it. You are one pass away from that on DefaultText — which is Lesson 5.

status · resources · mission