Naming the Flake
"Flaky" is not a diagnosis. It is the absence of one. This lesson gives you the vocabulary to replace it, and a diagnostic that tells you which word applies — without reading a single line of the failing test.
~15 min · unlocks A2: The Isolation Ladder
1. The split you're missing
You described a racy test as one with "resource contention with another test." That's half of it — and it's the half that doesn't dominate the literature.
Luo et al. classified 201 real flaky-test fixes from Apache projects. The two biggest categories are distinct in a way that matters enormously for how you fix them:
Async Wait
The test races the system under test's own asynchrony. One test, running completely alone, on an idle machine, can still fail. The test asserted before the app finished.
Fix shape: express the wait as a condition, not a duration.
Concurrency
The test races another test over shared mutable state. Passes alone, fails in parallel. This is the one you named.
Fix shape: remove the sharing, or serialise access to it.
Then a third, which is neither, and which people constantly misfile as one of the above:
Test Order Dependency (TOD)
Test B passes iff test A ran first (or didn't). Nothing is concurrent here at all — a purely sequential run can exhibit it. The state leaked forward in time rather than sideways across workers.
Fix shape: make each test set up what it needs and leave nothing behind.
Why the split matters for your sleep question. You said a passing sleep means "something else is changing an underlying resource concurrently." Usually it means something simpler and lonelier: the app itself hadn't finished. A sleep that fixes a test tells you only one thing — the test proceeded before the system reached the state it asserted on. It does not tell you who was responsible. That's why a sleep is a bad fix and a bad diagnostic: it papers over async-wait and concurrency identically, so you never learn which you had.
2. Glossary
Canonical terms. Use these instead of "flaky" and your bug reports become actionable.
| Race condition | Correctness depends on the relative timing of two operations, and that timing is not controlled. Note: not necessarily two threads — a test and an in-flight HTTP request race too. |
| Shared mutable state | The precondition for a concurrency bug. Remove either word — make it unshared, or make it immutable — and the bug is structurally impossible. |
| Critical section | A span of code that must not be interleaved with another actor's access to the same state. |
| Mutual exclusion (mutex) | The primitive that enforces one-at-a-time entry to a critical section. In Playwright, test.describe.serial and --workers=1 are coarse mutexes. |
| Isolation | Each actor behaves as if it were alone. The alternative to mutual exclusion, and almost always the better one: exclusion costs throughput, isolation doesn't. |
| Idempotence | Running an operation twice has the same effect as running it once. The property that makes a retry safe. |
| Atomicity | An operation is observable as either fully done or not started — never halfway. |
| Quiescence | The state of "no work in flight." What you actually want to wait for, and what you can rarely observe directly — hence polling. |
| Polling / auto-retrying assertion | Re-evaluate a condition until true or until a deadline. Playwright's web-first assertions. Converts "wait long enough" into "wait until true." |
| Poison state | Residue left by a failed test that causes subsequent tests to fail. The mechanism by which one bug becomes a scattered outage. |
| Tenant | The isolation boundary in a multi-tenant application — company, org, workspace. Whether your tests share one is the structural question for an E2E suite. |
| Fixture scope | The lifetime of a set-up value: test (fresh each test) or worker (shared by every test in one worker process). Scope is a sharing decision. |
3. Recall check
From memory, without scrolling up: name as many root-cause categories as you can. Then reveal and compare — the gap between what you wrote and the list is the thing worth re-reading.
The ten from Luo et al., roughly by frequency:
Async Wait · Concurrency · Test Order Dependency · Resource Leak · Network · Time · IO · Randomness · Floating Point · Unordered Collections
JavaScript suites add one the original paper under-weights: Platform — OS, browser version, font rendering, CPU count. It is disproportionately large in JS/TS.
You only need fluency in the top three plus Resource Leak and Unordered Collections. The others are rare and self-evident when they happen.
4. Bug hunt
Click the line that makes each test unreliable. One of these five is clean — for that one, click the "no bug" button.
These are shaped like your suite, not like a textbook.
5. Diagnosis from symptoms alone
You can classify most flakes before opening the test. Each symptom pattern below has a dominant explanation. Write yours, then reveal.
6. Retries: the honest rule
You said a green-on-retry is never acceptable. That instinct is right about fixes and wrong about instruments. The distinction:
Retry as instrument — legitimate
A retry that turns a hard failure into a labelled, counted, alarmed flaky result. The suite stays green so the team isn't blocked; the flaky count is a tracked number with a budget, and blowing the budget stops the line. You are trading immediate signal for aggregate signal, deliberately.
Retry as fix — a lie
A retry nobody counts. The badge is decoration. Failure rate per attempt can climb steadily and the dashboard stays green until it crosses the retry count and the suite falls over all at once.
The rule worth applying: retries are permitted only where the flaky count is a tracked metric with a ratchet. If nobody would notice the number doubling, the retries are load-bearing and you have no test suite — you have a coin flip with three tosses.
Applied to your config. retries: 3 means four attempts. If a test has an independent 50% failure rate, it reports green 94% of the time. Three retries doesn't reduce flakiness — it reduces your ability to perceive flakiness by roughly an order of magnitude. And note the interaction with poison state: if the failing test leaks data on attempt 1, attempts 2–4 run against a dirtier tenant than attempt 1 did. Retrying can make a leaky test less likely to pass, and can knock over the tests running beside it.
Trade-off summary
| Lever | Buys you | Costs you |
|---|---|---|
| More workers | Wall-clock | Exposes every shared-state bug at once |
--workers=1 | Kills concurrency flakes outright | Wall-clock; and TOD survives untouched |
describe.serial | Targeted exclusion | One failure skips the rest of the block; hides the real bug |
| Retries | Unblocked team | Signal loss ∝ retry count; can amplify poison state |
| Per-test isolation | Removes the precondition entirely | Setup cost per test; needs app or infra support |
| Longer timeouts | Absorbs slow environments | Perf regressions become invisible; failures get slow |
7. Your suite, classified
Three facts I established by reading apps/frontend/e2e:
- Zero
waitForTimeoutin 233 specs. Web-first assertions throughout,actionTimeoutandexpect.timeoutboth pinned at 5s. The Async Wait category is, by the standards of most suites, already solved. - Every worker in every project authenticates as one user, in one company.
auth.setup.tslogs inDEFAULT_USERonce and writes a singlestorageState; all eleven projects consume it. Your own comment says it plainly: "shared, JWT-bound company — there is no per-test company." - You already have guard fixtures.
companyNameGuardsnapshots and restores; theframe-types-seedcomment records a bug where worker-scoped seeding overflowed a virtualised list. These are not incidental — they are the scar tissue of a shared-tenant design.
And the symptom you reported: scattered, and equally bad locally and in CI. Run that through §5 and it excludes Platform (CI-only), excludes load-induced Async Wait (CI-worse), and excludes a single broken test (scattered). What's left is state shared across the tests themselves.
Your suite is not flaky because of how it waits. It is flaky because 233 specs share one mutable tenant — and retries: 3 is currently hiding roughly how badly.
That reframes the work. You do not have a scattering of unrelated timing bugs to grind through one at a time. You have one architectural property, and the flakes are its symptoms. That is much better news than it sounds.
Worth stating the limit of this claim: I established it from the suite's structure and your description of the failure texture, not from failure logs. The cheap confirmation is in "Retain this" below.
Retain this
Storage strength comes from retrieval and from use, not from re-reading. Two things, both cheap:
- Run the confirmation.
--workers=1 --retries=0on the whole suite, then--workers=4 --retries=0. If the first is near-clean and the second scatters, the diagnosis above is confirmed and you have a baseline number. If both fail in the same places, you have Test Order Dependency underneath as well — which is a different fix, and worth knowing before you start. - Next time a test fails, write the category name before you open the file. Commit to it, then check. Being wrong out loud is what builds the diagnostic; silently reading the taxonomy again does nothing.
The Isolation Ladder
Question 5 was your blank. There are seven rungs between "share everything" and "a tenant per test", each with a different price. A2 walks the ladder and places your resources on it.
Sources: Luo et al., FSE 2014 · Flaky Tests in JavaScript · RESOURCES.md · STATUS.html