Skip to content

Commit 083534d

Browse files
Chan Mengclaude
andcommitted
docs(testing): record the corpus sweep's blind spot, and file the load-sensitive case
Two findings from the 4.9 work, neither of which belongs in that item's own entry. docs/testing.md: the house corpus sweep — describe() + lint() + every storey's SVG, 95 artifacts — carries NO parse- or resolve-stage diagnostic, because lint() is the soundness layer alone. Measured: `arch lint materials` returns [] while `arch compile materials` returns W_SCALE_OVERFLOW. So a change to W_SCALE_OVERFLOW, W_DRAWING_OVERFLOW, any W_*_OFF_WALL or E_PARSE is invisible to a clean sweep, which then reports "nothing moved" in the same reassuring voice it uses when nothing did. Sweep 125 when a change can reach the resolver, and pick the non-vacuity plant with care: a lint-stage change moves nothing in the diagnostic sweep and would wrongly read as a broken check. Backlog 4.10: test/roof.test.ts's PDF case opens with a dynamic import of pdfkit under a 5000 ms budget. 674 ms alone, 5821 ms under heavy parallel load — it fails as a timeout there and passes 32/32 three times alone, including on main. Filed rather than smoothed over, because AGENTS.md records that this suite has no known flake. The entry notes it is the mirror of G.9: there the text was NOT a timeout and two agents assumed load anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDo6xNNG4fJpBAPR3ha8R4
1 parent e2a290a commit 083534d

2 files changed

Lines changed: 57 additions & 0 deletions

File tree

docs/backlog.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1747,6 +1747,38 @@ auto-fit cannot start seeing ground.
17471747
generator's size cap, and the full catalogue reaches a model through `llms-full.txt` /
17481748
`arch context --section errors` either way, which DID regenerate.
17491749

1750+
### 4.10 · A 5 s test budget on a heavy optional import — `todo`
1751+
1752+
Found 2026-09-04 while gating item 4.9, and worth filing because `AGENTS.md` currently records that
1753+
the suite has **no known flake**.
1754+
1755+
`test/roof.test.ts` → "PDF export draws it" opens with `await import("pdfkit")`, a heavy optional
1756+
dependency, under vitest's default **5000 ms** per-test budget. Measured on this machine:
1757+
1758+
| condition | duration | verdict |
1759+
|---|---|---|
1760+
| file run alone, machine idle | **674 ms** | passes, 7.4x margin |
1761+
| inside a full-suite run under heavy parallel load | **5821 ms** | **fails: `Test timed out in 5000ms`** |
1762+
1763+
So the margin is real but it is a margin on an **import**, not on the assertion — the same run had
1764+
`test/zones.test.ts` take 105 s. It reproduced twice under load and passed 32/32 three times when run
1765+
alone, including on `main`, so it is not a defect in `roof` and was not introduced by 4.9, which does
1766+
not touch that file.
1767+
1768+
**The trap this entry exists to mark.** The failure text says `Test timed out in 5000ms`, which is
1769+
exactly the diagnosis item G.9 records being WRONG about twice — there, the text said
1770+
`expected at least 1 JSON envelope(s) on stdout, saw 0`, which is not a timeout at all, and two agents
1771+
theorised about load from the *shape* of the symptom without reading it. Here the text really is a
1772+
timeout and the margin was measured, so the reading holds. **Read the assertion, then measure the
1773+
margin; do not pattern-match either way.**
1774+
1775+
Options, in the order they are worth trying: give this one case an explicit timeout that reflects what
1776+
it does (a dynamic import of an optional native-ish package is not a 5 ms unit test); or hoist the
1777+
capability probe so the import is paid once per file rather than inside the timed case; or leave it and
1778+
say plainly in `AGENTS.md` that the suite has one load-sensitive case, since a number nobody can
1779+
reproduce on a quiet machine is worse than a documented one. **Do not raise the global
1780+
`testTimeout`** — that hides every future instance of the same shape.
1781+
17501782
### 4.1 · Joinery pass performance — `todo`
17511783

17521784
`toScene` got roughly **3× slower** when v1.30 replaced the three wall-lowering paths with one

docs/testing.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,31 @@ than stripping text), and remember the corpus fans out: 30 `examples/*.arch` pro
449449
because `garden-house`, `hillside-villa`, `townhouse` and `two-storey` emit one per storey. A sweep
450450
that finds 30 has already lost the multi-storey plans.
451451

452+
**The house corpus sweep has a BLIND SPOT, and it is not small.** The sweep everyone reaches for is
453+
`describe()` + `lint()` + every storey's SVG — 95 artifacts over the 30 examples. Those three payloads
454+
between them carry **no parse- or resolve-stage diagnostic**, because `lint()` is the soundness layer
455+
alone. Measured:
456+
457+
```
458+
arch lint examples/materials.arch -> []
459+
arch compile examples/materials.arch -> ["W_SCALE_OVERFLOW"]
460+
```
461+
462+
So a change to `W_SCALE_OVERFLOW`, `W_DRAWING_OVERFLOW`, any `W_*_OFF_WALL`, `E_PARSE` or anything
463+
else raised before `lint()` runs is **invisible to a clean 95-artifact sweep**, and the sweep will
464+
report "nothing moved" in the same reassuring voice it uses when nothing did. Found in 2026-09 while
465+
closing backlog 4.9, whose whole subject is a resolve-stage diagnostic: the specified sweep could not
466+
see the item's own change.
467+
468+
**Sweep 125 when a change can reach the resolver**: add `compile().diagnostics` as a fourth payload,
469+
one row per example. The two sets are complements — `lint()` covers the lint rules and nothing else,
470+
`compile()` covers parse and resolve and nothing else — so neither alone is a corpus check.
471+
472+
Prove that one non-vacuous the same way, and pick the plant with care: a lint-stage change (repairing
473+
an example's furniture, say) moves **nothing** in the diagnostic sweep and would wrongly read as a
474+
broken check. Perturb something the resolver decides — moving `materials.arch` from A3 to A2 drops
475+
its `W_SCALE_OVERFLOW` and the sweep names exactly that row.
476+
452477
The same instinct applies to the code under test: **zero corpus movement is not evidence the new path
453478
ran.** Instrument it with a counter, show which examples reach it and how often, then remove the
454479
counter and confirm with `grep -c`. Backlog G.1 is the worked example — the changed path fires 228

0 commit comments

Comments
 (0)