Skip to content

Commit 38dc5af

Browse files
committed
docs(adrs): add ARCH-025 for idiomatic bun:test parametrization and matchers
Codifies two anti-patterns found across 24 test files during this session's audit: manual loops standing in for test.each()/ describe.each(), and generic boolean assertions (expect(x).toBe(true)) instead of specific matchers. Kept separate from ARCH-005, which is already at its review-context briefing-budget ceiling. Automated enforcement (a candidate oxlint plugin) is not yet implemented; rules is false and manual review is the enforcement layer for now. Signed-off-by: Rhuan Barreto <rhuan@barreto.work>
1 parent ba6f480 commit 38dc5af

1 file changed

Lines changed: 138 additions & 0 deletions

File tree

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
id: ARCH-025
3+
title: Idiomatic bun:test Parametrization and Matchers
4+
domain: architecture
5+
rules: false
6+
files:
7+
- "tests/**/*.ts"
8+
---
9+
10+
## Context
11+
12+
ARCH-005 governs test isolation, lifecycle, and hygiene (temp directories, spy/env restoration, assertion presence) but does not prescribe how a test author should express "the same check against many inputs" or "a derived true/false fact." Bun's own testing guidance (<https://bun.sh/docs/test/writing-tests>) is prescriptive on both points, and an audit of every file under `tests/**/*.test.ts` in this repository found the two gaps recurring widely enough to be a convention problem rather than isolated mistakes:
13+
14+
1. **Manual loops standing in for `test.each()`/`describe.each()`.** 32 confirmed instances across 8 files — for example `tests/formats/project-config-fuzz.test.ts` (12 instances) and `tests/engine/rule-scanner-escapes.test.ts` (8 instances) each looped over an array of cases and either called `test()` inside the loop body to register cases dynamically, or called `expect()` once per item inside a single test for logically independent scenarios. Bun's docs name this exact shape — "manual loops instead of `test.each()`" — as an anti-pattern.
15+
2. **Generic boolean assertions instead of specific matchers.** 28 confirmed instances across 16 files — `expect(x === y).toBe(true)`, `expect(arr.some(...)).toBe(true)`, `expect(Array.isArray(x)).toBe(true)`, `expect(arr.every(...)).toBe(true)`. Bun's docs explicitly recommend matchers such as `.toHaveLength()`, `.toContain()`, and `.toBeGreaterThanOrEqual()` over wrapping a derived boolean in `.toBe(true)`/`.toBe(false)`.
16+
17+
Both patterns compile, pass `oxlint`, and pass `bun test` — they are not caught by any existing automated check, which is exactly why they spread undetected across 24 independently-written files. They also share a common cost: a manual loop reports one pass/fail for N cases, so a regression in case 3 of 12 is invisible in the test summary and must be found by reading the loop body; a boolean-collapsed assertion reports `expected true, got false` with no indication of which value or array element was actually wrong.
18+
19+
**Alternatives considered:**
20+
21+
- **Add these as more Do's/Don'ts to ARCH-005** — Rejected: ARCH-005 is already 199 lines and its own Compliance section documents that its Do's and Don'ts already exceed the `archgate review-context` briefing budget (2000-char cap); appending more would push more of it out of every future briefing, including the parts already there.
22+
- **Enforce via a new oxlint plugin immediately** — Deferred, not rejected: this repo already has a precedent (`lint/expect-expect.ts`, `lint/no-bare-env-restore.ts`) for exactly this kind of AST-detectable test-shape rule, and it is the right enforcement layer for both patterns (syntax-detectable, not behavioral). Building it is out of scope for the session that produced this ADR, which only fixed existing instances; the rule is recorded as future work in Compliance and Enforcement below.
23+
- **Leave undocumented, rely on code review alone** — Rejected: the instances this ADR responds to were already written by reviewed, merged PRs across 24 files; undocumented convention did not prevent the drift.
24+
25+
## Decision
26+
27+
Tests under `tests/**/*.ts` MUST express "the same assertion logic against multiple inputs" with `test.each()`/`describe.each()`, and MUST assert derived facts with the most specific matcher available rather than collapsing a comparison into a boolean passed to `.toBe(true)`/`.toBe(false)`.
28+
29+
**Scope:** This ADR covers test-case parametrization and matcher choice only. It does not cover test isolation, lifecycle hooks, or fixture/temp-directory handling — those remain governed by ARCH-005.
30+
31+
A loop is "standing in for `test.each()`" when either:
32+
33+
- its body calls `test()`, `it()`, or `describe()` to register a case dynamically, or
34+
- it calls `expect()` once per iteration inside a single test, for items that represent logically independent scenarios (each could fail or pass independently of the others).
35+
36+
A loop that only builds shared setup data or fixtures for a single overall assertion is not covered by this decision.
37+
38+
## Do's and Don'ts
39+
40+
### Do
41+
42+
- **DO** use `test.each(cases)("<title with %s/%p/%d>", (...args) => { ... })` when the same check runs against an array of independent inputs, so each case is its own named, independently-reportable test.
43+
- **DO** use `describe.each(cases)(...)` when each case needs its own group of multiple related tests, not just one assertion.
44+
- **DO** pass array rows (`[a, b, expected]`) for positional destructuring and object rows (`{a, b, expected}`) when the case is easier to read as named fields (format the title with `$field`).
45+
- **DO** assert a derived comparison directly on the values being compared: `expect(actual).toBe(expected)`, `expect(actual).toEqual(expected)`.
46+
- **DO** use the matcher that matches the shape of the check: `.toContain()`/`.toMatch()` for substrings, `.toBeInstanceOf()` for type checks (including `Array.isArray` replacements), `.toHaveLength()` for counts, `.find(...) `+ `.toBeDefined()`/`.toBeUndefined()` for "does at least one item satisfy X."
47+
- **DO** keep the loop→`test.each()` conversion 1:1 — every assertion that ran per-iteration in the original loop must still run per-case in the converted version; a conversion MUST NOT silently drop or merge assertions.
48+
49+
### Don't
50+
51+
- **DON'T** write `for (const case of cases) { test(...); }` or `for (const case of cases) { expect(...); }` inside a single test — use `test.each()`/`describe.each()` instead.
52+
- **DON'T** write `expect(a === b).toBe(true)` or `expect(a === b).toBe(false)` — assert `expect(a).toBe(b)` / `expect(a).not.toBe(b)` directly.
53+
- **DON'T** write `expect(arr.some(predicate)).toBe(true)` or `expect(arr.every(predicate)).toBe(true)` — these collapse an array-wide check into an opaque boolean that reports `false !== true` on failure with no indication of which element (or none) satisfied the predicate. Use `.find(predicate)` + `.toBeDefined()`, a per-item loop of `expect(item).toBe(...)` calls, or a matcher over the mapped values (e.g. `expect(arr.map(x => x.message)).toContain(...)`).
54+
- **DON'T** write `expect(Array.isArray(x)).toBe(true)` — use `expect(x).toBeInstanceOf(Array)`.
55+
- **DON'T** precompute a boolean in a local variable purely to assert it (`const equal = JSON.stringify(a) === JSON.stringify(b); expect(equal).toBe(true);`) — assert on `a` and `b` (or their comparable projections) directly with `.toEqual()`/`.toBe()` so a failure shows the actual diff.
56+
57+
## Implementation Pattern
58+
59+
### Good Example
60+
61+
```typescript
62+
// tests/engine/rule-scanner.test.ts
63+
const bannedModules = ["node:fs", "fs", "node:child_process", "child_process"];
64+
65+
test.each(bannedModules)("blocks %s import", (mod) => {
66+
const violations = scanRuleSource(`import x from "${mod}";`);
67+
expect(violations.length).toBeGreaterThan(0);
68+
});
69+
70+
test("reports a specific banned-global message", () => {
71+
const violations = scanRuleSource(`fetch("https://x")`);
72+
const match = violations.find((v) => v.message.includes('"fetch" global'));
73+
expect(match).toBeDefined();
74+
});
75+
```
76+
77+
### Bad Example
78+
79+
```typescript
80+
// BAD: a manual loop registers one test per module — a regression in any
81+
// single module's assertion doesn't produce a per-module reported failure
82+
// the same way test.each() would, and adding/removing a case means editing
83+
// loop internals rather than a data table.
84+
for (const mod of ["node:fs", "fs", "node:child_process", "child_process"]) {
85+
test(`blocks ${mod} import`, () => {
86+
const violations = scanRuleSource(`import x from "${mod}";`);
87+
expect(violations.length).toBeGreaterThan(0);
88+
});
89+
}
90+
91+
// BAD: collapses "does any violation mention fetch" into an opaque boolean —
92+
// on failure this reports "expected true, got false" with no indication of
93+
// what the violations actually were.
94+
expect(violations.some((v) => v.message.includes('"fetch" global'))).toBe(true);
95+
```
96+
97+
## Consequences
98+
99+
### Positive
100+
101+
- **Per-case failure reporting**: `test.each()` gives each input its own named test result, so a regression in one case is immediately identified by name instead of requiring a developer to read a loop body to find which iteration failed.
102+
- **Readable failure diffs**: asserting on the actual compared values (instead of a boolean) means a failing test's output shows what was expected vs. what was received, cutting debugging time.
103+
- **Data-as-table**: adding or removing a case becomes a one-line change to an array/object literal rather than an edit to loop logic.
104+
105+
### Negative
106+
107+
- **More verbose for one-off checks**: a single ad-hoc assertion inside `.some()`/`.every()` is sometimes fewer characters than the specific-matcher equivalent; this ADR accepts that verbosity in exchange for diagnosability.
108+
- **`test.each()` output naming requires care**: a poorly chosen format string (e.g. always `%s` regardless of row shape) produces indistinguishable test names across cases, which undoes the per-case reporting benefit this decision is meant to provide.
109+
110+
### Risks
111+
112+
- **No automated enforcement yet**: neither pattern is currently caught by `oxlint` or `archgate check`, so new instances can still be introduced and merged undetected, the same way the original 60 were.
113+
- **Mitigation:** tracked as future work in Compliance and Enforcement below (a candidate `oxlint` plugin following the existing `bun-test/expect-expect` precedent); until it exists, code review is the only enforcement layer, so reviewers MUST check new/changed test files against the Do's and Don'ts above.
114+
- **Existing violations elsewhere in the codebase may resurface**: this ADR's Context reflects a point-in-time audit of 120 files; files not covered by that audit may still contain either anti-pattern.
115+
- **Mitigation:** treat any test file touched for an unrelated change as an opportunity to fix nearby instances of either pattern, consistent with ARCH-005's existing test-hygiene expectations.
116+
117+
## Compliance and Enforcement
118+
119+
### Automated Enforcement
120+
121+
- **Not yet implemented.** No `oxlint` rule or `archgate` rule currently detects either anti-pattern. A future `oxlint` plugin (e.g. `bun-test/no-loop-test-cases` and `bun-test/no-boolean-collapse`, registered via `jsPlugins` in `.oxlintrc.json` alongside the existing `bun-test/expect-expect` and `test-isolation/no-bare-env-restore` plugins) is the intended enforcement layer once written, matching this repo's existing pattern of using oxlint for syntax-detectable test-shape rules and reserving `archgate` rules for structural/governance checks. This ADR's `rules: false` reflects that no companion `.rules.ts` exists.
122+
123+
### Manual Enforcement
124+
125+
Code reviewers MUST verify, for any new or changed file under `tests/**/*.ts`:
126+
127+
1. No `for`/`.forEach` loop registers a `test()`/`it()`/`describe()` call, and no `for`/`.forEach` loop inside a single test calls `expect()` once per logically independent case — either shape MUST be `test.each()`/`describe.each()` instead.
128+
2. No `expect(<comparison>).toBe(true)` or `.toBe(false)` where `<comparison>` is itself a boolean expression (`===`, `.some()`, `.every()`, `Array.isArray()`, `.includes()`) — the assertion MUST target the underlying values with a specific matcher.
129+
3. A `test.each()`/`describe.each()` conversion preserves every assertion that ran in the original loop — none dropped, none merged into a single case.
130+
131+
### Exceptions
132+
133+
None currently approved. A test file with a genuine reason to keep a loop (e.g. a fuzz/property check verifying one invariant across many generated inputs, where the inputs are not independently meaningful test cases) is not a violation of this ADR in the first place — see the Decision section's definition of what counts as "standing in for `test.each()`."
134+
135+
## References
136+
137+
- [ARCH-005 — Testing Standards](./ARCH-005-testing-standards.md) — governs test isolation, lifecycle hooks, and assertion presence; this ADR covers parametrization and matcher choice specifically and does not restate ARCH-005's conventions.
138+
- [Bun test runner — Writing Tests](https://bun.sh/docs/test/writing-tests) — source of the `test.each()`/`describe.each()` API and the anti-pattern guidance this ADR codifies.

0 commit comments

Comments
 (0)