|
| 1 | +--- |
| 2 | +id: ARCH-025 |
| 3 | +title: Idiomatic bun:test Parametrization and Matchers |
| 4 | +domain: architecture |
| 5 | +rules: false |
| 6 | +files: |
| 7 | + - "tests/**/*.ts" |
| 8 | +--- |
| 9 | + |
| 10 | +## Context |
| 11 | + |
| 12 | +ARCH-005 governs test isolation, lifecycle, and hygiene (temp directories, spy/env restoration, assertion presence) but does not prescribe how a test author should express "the same check against many inputs" or "a derived true/false fact." Bun's own testing guidance (<https://bun.sh/docs/test/writing-tests>) is prescriptive on both points, and an audit of every file under `tests/**/*.test.ts` in this repository found the two gaps recurring widely enough to be a convention problem rather than isolated mistakes: |
| 13 | + |
| 14 | +1. **Manual loops standing in for `test.each()`/`describe.each()`.** 32 confirmed instances across 8 files — for example `tests/formats/project-config-fuzz.test.ts` (12 instances) and `tests/engine/rule-scanner-escapes.test.ts` (8 instances) each looped over an array of cases and either called `test()` inside the loop body to register cases dynamically, or called `expect()` once per item inside a single test for logically independent scenarios. Bun's docs name this exact shape — "manual loops instead of `test.each()`" — as an anti-pattern. |
| 15 | +2. **Generic boolean assertions instead of specific matchers.** 28 confirmed instances across 16 files — `expect(x === y).toBe(true)`, `expect(arr.some(...)).toBe(true)`, `expect(Array.isArray(x)).toBe(true)`, `expect(arr.every(...)).toBe(true)`. Bun's docs explicitly recommend matchers such as `.toHaveLength()`, `.toContain()`, and `.toBeGreaterThanOrEqual()` over wrapping a derived boolean in `.toBe(true)`/`.toBe(false)`. |
| 16 | + |
| 17 | +Both patterns compile, pass `oxlint`, and pass `bun test` — they are not caught by any existing automated check, which is exactly why they spread undetected across 24 independently-written files. They also share a common cost: a manual loop reports one pass/fail for N cases, so a regression in case 3 of 12 is invisible in the test summary and must be found by reading the loop body; a boolean-collapsed assertion reports `expected true, got false` with no indication of which value or array element was actually wrong. |
| 18 | + |
| 19 | +**Alternatives considered:** |
| 20 | + |
| 21 | +- **Add these as more Do's/Don'ts to ARCH-005** — Rejected: ARCH-005 is already 199 lines and its own Compliance section documents that its Do's and Don'ts already exceed the `archgate review-context` briefing budget (2000-char cap); appending more would push more of it out of every future briefing, including the parts already there. |
| 22 | +- **Enforce via a new oxlint plugin immediately** — Deferred, not rejected: this repo already has a precedent (`lint/expect-expect.ts`, `lint/no-bare-env-restore.ts`) for exactly this kind of AST-detectable test-shape rule, and it is the right enforcement layer for both patterns (syntax-detectable, not behavioral). Building it is out of scope for the session that produced this ADR, which only fixed existing instances; the rule is recorded as future work in Compliance and Enforcement below. |
| 23 | +- **Leave undocumented, rely on code review alone** — Rejected: the instances this ADR responds to were already written by reviewed, merged PRs across 24 files; undocumented convention did not prevent the drift. |
| 24 | + |
| 25 | +## Decision |
| 26 | + |
| 27 | +Tests under `tests/**/*.ts` MUST express "the same assertion logic against multiple inputs" with `test.each()`/`describe.each()`, and MUST assert derived facts with the most specific matcher available rather than collapsing a comparison into a boolean passed to `.toBe(true)`/`.toBe(false)`. |
| 28 | + |
| 29 | +**Scope:** This ADR covers test-case parametrization and matcher choice only. It does not cover test isolation, lifecycle hooks, or fixture/temp-directory handling — those remain governed by ARCH-005. |
| 30 | + |
| 31 | +A loop is "standing in for `test.each()`" when either: |
| 32 | + |
| 33 | +- its body calls `test()`, `it()`, or `describe()` to register a case dynamically, or |
| 34 | +- it calls `expect()` once per iteration inside a single test, for items that represent logically independent scenarios (each could fail or pass independently of the others). |
| 35 | + |
| 36 | +A loop that only builds shared setup data or fixtures for a single overall assertion is not covered by this decision. |
| 37 | + |
| 38 | +## Do's and Don'ts |
| 39 | + |
| 40 | +### Do |
| 41 | + |
| 42 | +- **DO** use `test.each(cases)("<title with %s/%p/%d>", (...args) => { ... })` when the same check runs against an array of independent inputs, so each case is its own named, independently-reportable test. |
| 43 | +- **DO** use `describe.each(cases)(...)` when each case needs its own group of multiple related tests, not just one assertion. |
| 44 | +- **DO** pass array rows (`[a, b, expected]`) for positional destructuring and object rows (`{a, b, expected}`) when the case is easier to read as named fields (format the title with `$field`). |
| 45 | +- **DO** assert a derived comparison directly on the values being compared: `expect(actual).toBe(expected)`, `expect(actual).toEqual(expected)`. |
| 46 | +- **DO** use the matcher that matches the shape of the check: `.toContain()`/`.toMatch()` for substrings, `.toBeInstanceOf()` for type checks (including `Array.isArray` replacements), `.toHaveLength()` for counts, `.find(...) `+ `.toBeDefined()`/`.toBeUndefined()` for "does at least one item satisfy X." |
| 47 | +- **DO** keep the loop→`test.each()` conversion 1:1 — every assertion that ran per-iteration in the original loop must still run per-case in the converted version; a conversion MUST NOT silently drop or merge assertions. |
| 48 | + |
| 49 | +### Don't |
| 50 | + |
| 51 | +- **DON'T** write `for (const case of cases) { test(...); }` or `for (const case of cases) { expect(...); }` inside a single test — use `test.each()`/`describe.each()` instead. |
| 52 | +- **DON'T** write `expect(a === b).toBe(true)` or `expect(a === b).toBe(false)` — assert `expect(a).toBe(b)` / `expect(a).not.toBe(b)` directly. |
| 53 | +- **DON'T** write `expect(arr.some(predicate)).toBe(true)` or `expect(arr.every(predicate)).toBe(true)` — these collapse an array-wide check into an opaque boolean that reports `false !== true` on failure with no indication of which element (or none) satisfied the predicate. Use `.find(predicate)` + `.toBeDefined()`, a per-item loop of `expect(item).toBe(...)` calls, or a matcher over the mapped values (e.g. `expect(arr.map(x => x.message)).toContain(...)`). |
| 54 | +- **DON'T** write `expect(Array.isArray(x)).toBe(true)` — use `expect(x).toBeInstanceOf(Array)`. |
| 55 | +- **DON'T** precompute a boolean in a local variable purely to assert it (`const equal = JSON.stringify(a) === JSON.stringify(b); expect(equal).toBe(true);`) — assert on `a` and `b` (or their comparable projections) directly with `.toEqual()`/`.toBe()` so a failure shows the actual diff. |
| 56 | + |
| 57 | +## Implementation Pattern |
| 58 | + |
| 59 | +### Good Example |
| 60 | + |
| 61 | +```typescript |
| 62 | +// tests/engine/rule-scanner.test.ts |
| 63 | +const bannedModules = ["node:fs", "fs", "node:child_process", "child_process"]; |
| 64 | + |
| 65 | +test.each(bannedModules)("blocks %s import", (mod) => { |
| 66 | + const violations = scanRuleSource(`import x from "${mod}";`); |
| 67 | + expect(violations.length).toBeGreaterThan(0); |
| 68 | +}); |
| 69 | + |
| 70 | +test("reports a specific banned-global message", () => { |
| 71 | + const violations = scanRuleSource(`fetch("https://x")`); |
| 72 | + const match = violations.find((v) => v.message.includes('"fetch" global')); |
| 73 | + expect(match).toBeDefined(); |
| 74 | +}); |
| 75 | +``` |
| 76 | + |
| 77 | +### Bad Example |
| 78 | + |
| 79 | +```typescript |
| 80 | +// BAD: a manual loop registers one test per module — a regression in any |
| 81 | +// single module's assertion doesn't produce a per-module reported failure |
| 82 | +// the same way test.each() would, and adding/removing a case means editing |
| 83 | +// loop internals rather than a data table. |
| 84 | +for (const mod of ["node:fs", "fs", "node:child_process", "child_process"]) { |
| 85 | + test(`blocks ${mod} import`, () => { |
| 86 | + const violations = scanRuleSource(`import x from "${mod}";`); |
| 87 | + expect(violations.length).toBeGreaterThan(0); |
| 88 | + }); |
| 89 | +} |
| 90 | + |
| 91 | +// BAD: collapses "does any violation mention fetch" into an opaque boolean — |
| 92 | +// on failure this reports "expected true, got false" with no indication of |
| 93 | +// what the violations actually were. |
| 94 | +expect(violations.some((v) => v.message.includes('"fetch" global'))).toBe(true); |
| 95 | +``` |
| 96 | + |
| 97 | +## Consequences |
| 98 | + |
| 99 | +### Positive |
| 100 | + |
| 101 | +- **Per-case failure reporting**: `test.each()` gives each input its own named test result, so a regression in one case is immediately identified by name instead of requiring a developer to read a loop body to find which iteration failed. |
| 102 | +- **Readable failure diffs**: asserting on the actual compared values (instead of a boolean) means a failing test's output shows what was expected vs. what was received, cutting debugging time. |
| 103 | +- **Data-as-table**: adding or removing a case becomes a one-line change to an array/object literal rather than an edit to loop logic. |
| 104 | + |
| 105 | +### Negative |
| 106 | + |
| 107 | +- **More verbose for one-off checks**: a single ad-hoc assertion inside `.some()`/`.every()` is sometimes fewer characters than the specific-matcher equivalent; this ADR accepts that verbosity in exchange for diagnosability. |
| 108 | +- **`test.each()` output naming requires care**: a poorly chosen format string (e.g. always `%s` regardless of row shape) produces indistinguishable test names across cases, which undoes the per-case reporting benefit this decision is meant to provide. |
| 109 | + |
| 110 | +### Risks |
| 111 | + |
| 112 | +- **No automated enforcement yet**: neither pattern is currently caught by `oxlint` or `archgate check`, so new instances can still be introduced and merged undetected, the same way the original 60 were. |
| 113 | + - **Mitigation:** tracked as future work in Compliance and Enforcement below (a candidate `oxlint` plugin following the existing `bun-test/expect-expect` precedent); until it exists, code review is the only enforcement layer, so reviewers MUST check new/changed test files against the Do's and Don'ts above. |
| 114 | +- **Existing violations elsewhere in the codebase may resurface**: this ADR's Context reflects a point-in-time audit of 120 files; files not covered by that audit may still contain either anti-pattern. |
| 115 | + - **Mitigation:** treat any test file touched for an unrelated change as an opportunity to fix nearby instances of either pattern, consistent with ARCH-005's existing test-hygiene expectations. |
| 116 | + |
| 117 | +## Compliance and Enforcement |
| 118 | + |
| 119 | +### Automated Enforcement |
| 120 | + |
| 121 | +- **Not yet implemented.** No `oxlint` rule or `archgate` rule currently detects either anti-pattern. A future `oxlint` plugin (e.g. `bun-test/no-loop-test-cases` and `bun-test/no-boolean-collapse`, registered via `jsPlugins` in `.oxlintrc.json` alongside the existing `bun-test/expect-expect` and `test-isolation/no-bare-env-restore` plugins) is the intended enforcement layer once written, matching this repo's existing pattern of using oxlint for syntax-detectable test-shape rules and reserving `archgate` rules for structural/governance checks. This ADR's `rules: false` reflects that no companion `.rules.ts` exists. |
| 122 | + |
| 123 | +### Manual Enforcement |
| 124 | + |
| 125 | +Code reviewers MUST verify, for any new or changed file under `tests/**/*.ts`: |
| 126 | + |
| 127 | +1. No `for`/`.forEach` loop registers a `test()`/`it()`/`describe()` call, and no `for`/`.forEach` loop inside a single test calls `expect()` once per logically independent case — either shape MUST be `test.each()`/`describe.each()` instead. |
| 128 | +2. No `expect(<comparison>).toBe(true)` or `.toBe(false)` where `<comparison>` is itself a boolean expression (`===`, `.some()`, `.every()`, `Array.isArray()`, `.includes()`) — the assertion MUST target the underlying values with a specific matcher. |
| 129 | +3. A `test.each()`/`describe.each()` conversion preserves every assertion that ran in the original loop — none dropped, none merged into a single case. |
| 130 | + |
| 131 | +### Exceptions |
| 132 | + |
| 133 | +None currently approved. A test file with a genuine reason to keep a loop (e.g. a fuzz/property check verifying one invariant across many generated inputs, where the inputs are not independently meaningful test cases) is not a violation of this ADR in the first place — see the Decision section's definition of what counts as "standing in for `test.each()`." |
| 134 | + |
| 135 | +## References |
| 136 | + |
| 137 | +- [ARCH-005 — Testing Standards](./ARCH-005-testing-standards.md) — governs test isolation, lifecycle hooks, and assertion presence; this ADR covers parametrization and matcher choice specifically and does not restate ARCH-005's conventions. |
| 138 | +- [Bun test runner — Writing Tests](https://bun.sh/docs/test/writing-tests) — source of the `test.each()`/`describe.each()` API and the anti-pattern guidance this ADR codifies. |
0 commit comments