Hierarchies in Backend Design · Lesson 5

Eventual consistency between aggregates

Lesson 2 said cross-level rules are enforced "eventually, by events and reconciliation". That was a promissory note. This lesson pays it.

1. The dual-write problem

Two resources, no transaction

You need to do two things: change your database, and tell someone else. Publish the campaign, and call Meta. Mark the subtree deleted, and update the search index. There is no transaction spanning both resources, so pick your poison:

Why no ordering fixes it

This is the dual-write problem, and the important thing to internalise is that there is no ordering that fixes it. Retries don't fix it either — a retry cannot run if the process is gone. Any code shaped like "await db.commit() … await somethingElse()" has this bug, and it is invisible in testing because it needs a crash in a specific window.

2. The transactional outbox

Reduce two resources to one

The transactional outbox reduces two resources to one. Write the state change and the intent-to-publish into the same database, in the same transaction. A separate process — the relay — reads the outbox table and publishes.

outbox(
  id            uuid PRIMARY KEY,     -- the dedup token for consumers
  seq           bigserial,            -- relay ordering
  aggregate_id  uuid NOT NULL,        -- partition key: per-aggregate ordering
  topic         text NOT NULL,
  body          jsonb NOT NULL,
  published_at  timestamptz)          -- NULL = not yet relayed

Atomicity is now the database's problem, which it is good at.

Two relay flavours

The relay can be polling (WHERE published_at IS NULL ORDER BY seq — simple, adds latency and load) or change data capture (tail the write-ahead log with something like Debezium — lower latency, more infrastructure). Start with polling.

Notice what this is: the outbox row is derived data whose truth is the state change — the same relationship as path to parent_id in Lesson 1 §5. That is why losing the broker is survivable and losing the transaction is not.

3. At-least-once, therefore idempotent consumers

The relay can publish and then die before marking the row — so the row is published again. Two relay instances can read the same batch. This gives you at-least-once delivery, and you should stop hoping for exactly-once: it isn't available across a network without making the consumer idempotent anyway, at which point at-least-once plus idempotency is the implementation.

The inbox pattern

So every consumer needs a dedup mechanism, keyed on the event id. The standard one is the inbox pattern — a table of processed event ids, written in the same transaction as the consumer's own state change:

BEGIN;
  INSERT INTO processed_events (event_id) VALUES ($1);   -- PK violation ⇒ already handled, roll back
  -- … apply the effect …
COMMIT;

Same shape as the idempotency key from Lesson 3 §4: a unique constraint is what makes check-then-act safe. Third appearance of that idea; it is the single most reusable thing in this path.

4. Ordering — and the hierarchy-specific way it bites

Per-aggregate ordering is what you need

Global ordering across a broker is expensive and almost never needed. Per-aggregate ordering usually is: NodeMoved then NodeDeleted applied in reverse leaves a deleted node visibly moved. Get it by partitioning on aggregate_id so one aggregate's events land in one ordered partition, and relaying in seq order.

The child-before-parent problem

Hierarchies add a failure mode that per-aggregate ordering does not cover, because each node is its own aggregate: a child's event can arrive before its parent's. Your search-index consumer receives NodeCreated(child, parentId: p) when it has never heard of p. Three responses, in increasing order of quality:

• Crash and retry — works if the parent event is genuinely coming, wastes throughput, and turns a permanently-missing parent into an infinite loop.
• Buffer the orphan until its parent appears, with a timeout to a dead-letter queue.
• Design the consumer to tolerate orphans — store the edge, render what you can, resolve later.

The third is the same relaxation of referential integrity your original notes proposed for the write path in Lesson 2 §3 — except here it is not optional. A consumer of a hierarchy's events must tolerate temporarily dangling parent references, because there is no transaction spanning the producers. If you were looking for the honest answer to "should we allow violations of referential integrity?", it is: inside one aggregate, never; across aggregates, you have no choice.

5. Enforcing the rule that spans levels

Back to Lesson 2's example: the sum of ad set budgets must not exceed the campaign budget, with Campaign and AdSet as separate aggregates. The rule cannot be a transactional invariant. So it becomes a pipeline:

AdSetBudgetChanged  →  handler recomputes the campaign's total
                    →  if over: emit CampaignOverBudget, set campaign.flagged = true,
                       pause delivery on the newest ad sets   ← compensating action

Three questions before you ship an eventual rule

A compensating action is the deliberate response to a violation you allowed to happen. Its existence is the price of the boundary, and specifying it is the design work people skip. Three questions you must answer before shipping an eventually-consistent rule — and if you cannot answer them, the rule is a true invariant after all and belongs inside one aggregate:

1. How long is the window? Not "eventually" — a number. Seconds, or a nightly batch. This is a product decision with a monetary cost when the rule is about money.
2. Who sees the violation, and what do they see? A user watching the campaign page during the window sees an over-budget campaign. Does the UI say "checking…", or show it as valid?
3. Who repairs it? Vernon's heuristic is the sharpest tool here: if the fix is a human's job, the rule is eventual; if the system must refuse the change outright, it is a true invariant.

6. Reconciliation, because pipelines leak

Events get dropped, consumers get deployed with bugs, brokers get purged during incidents. Every eventually-consistent system needs a periodic reconciliation sweep that recomputes the derived state from the source of truth and repairs the difference — and, importantly, reports the size of the difference, because a drift metric trending upward is your only early warning that the pipeline is broken.

The sweep for a hierarchy

For hierarchies the sweep is usually cheap and worth having on day one: recompute every node's path from parent_id, rebuild the closure table, re-derive each campaign's budget total. All of it is possible precisely because Lesson 1's rule was obeyed — the derived structures are rebuildable from a truth the database enforces. The outbox, the search index, the read model and the closure table are all the same kind of thing, and reconciliation is the same job for each.

If you only keep one sentence from this path: choose one source of truth the database enforces, and make everything else rebuildable from it. Storage models, aggregate boundaries, outboxes and reconciliation are all consequences of taking that seriously.

Exercises

Exercise 1 — Bug hunt: click the offending line

async function publishCampaign(user: User, id: string) {  const campaign = await db.campaign.findFirstOrThrow({ where: { id, tenantId: user.tenantId } });  await assertValid(campaign);  await db.campaign.update({ where: { id }, data: { status: 'PUBLISHED', publishedAt: new Date() } });  await meta.publishCampaign(campaign);  await searchIndex.upsert(campaign);  return { ok: true };}

Exercise 2 — Bug hunt: click the offending line (or declare it clean)

The polling relay.

async function relayBatch() {  await db.$transaction(async (tx) => {    await tx.$executeRaw`SELECT pg_advisory_xact_lock(hashtext('outbox-relay'))`;    const batch = await tx.outbox.findMany({      where: { publishedAt: null }, orderBy: { seq: 'asc' }, take: 100 });    for (const ev of batch) {      await broker.publish(ev.topic, { key: ev.aggregateId, id: ev.id, body: ev.body });      await tx.outbox.update({ where: { id: ev.id }, data: { publishedAt: new Date() } });    }  });}

Exercise 3 — Recall (free)

From memory: why does no ordering of "write to the database" and "call the external system" fix the dual-write problem? Then state what the outbox actually changes about the situation.

Exercise 4 — Execution (free)

Specify the enforcement of "sum of ad set budgets ≤ campaign budget" with Campaign and AdSet as separate aggregates. Answer all four: (a) what triggers the check, (b) the compensating action, (c) the window and what a user sees during it, (d) how you detect the pipeline silently failing.

Retention

• Recognise the shape: any await db.commit() followed by await externalCall() is a dual write. That grep is worth running on real codebases.
• The three-times idea: unique constraints are what make check-then-act safe — idempotency keys (L3), the inbox table (L5), and the closure-table primary key (L1). When you see check-then-act, ask what constraint backs it.
• Tomorrow, all five lessons: for each of storage model, aggregate boundary, API verb, move, and event pipeline, write the one question that decides it. Five questions on one page is the artefact you actually want in a design review.