Skip to content

Commit 769a647

Browse files
mfornetclaude
andcommitted
testsuite: correct stale command-execution and exit-code docstrings
Follow-up to #130, fixing docstring inaccuracies that the same cleanup left in place. Behaviour unchanged — comments only. - The module docstring claimed "Only `module`, `assert_return`, `assert_trap`, and `action` are executed". `register` (executed since it records a binding) and `assert_exception` are also executed; list them. - It listed `assert_invalid`/`assert_malformed` under "reported as `Skipped(<kind>)`", but those are executed: we decode and validate the module, `Pass` on correct rejection, and `Skipped(<kind>: not rejected)` only on wrongful acceptance. This also matches the `skippedBucket` comment #130 added. - The "Exit code: nonzero iff any `Fail`/`InterpreterError`/`OutOfFuel`" line ignored the exit-code-3 setup-failure paths (bad args, unreadable testsuite dir, no matching files, temp-dir failure). Spell out both codes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5c40dc9 commit 769a647

1 file changed

Lines changed: 12 additions & 7 deletions

File tree

interpreter/Interpreter/Testsuite.lean

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,19 +12,24 @@ contains `PATTERN` as a substring (case-sensitive). Subdirectories of
1212
1313
Per `.wast` file we shell out to `wasm-tools json-from-wast` to split the
1414
script into a JSON manifest plus per-module `.wasm` files, then walk the
15-
commands. Only `module`, `assert_return`, `assert_trap`, and `action` are
16-
executed — every other command type (`assert_invalid`, `assert_malformed`,
17-
`assert_unlinkable`, etc.) is reported as `Skipped(<kind>)`.
15+
commands. `module`, `register`, `assert_return`, `assert_trap`,
16+
`assert_exception`, and `action` are executed against the interpreter.
17+
`assert_invalid`/`assert_malformed` are also run — we decode and statically
18+
validate the module, pass when it is rejected, and record
19+
`Skipped(<kind>: not rejected)` only if we wrongly accept it. Every remaining
20+
command type (`assert_unlinkable`, etc.) falls through to `Skipped(<kind>)`.
1821
1922
Output is the human-readable per-file report by default. `--json` emits the
2023
results as a JSON array instead; `--report` emits the stable text coverage
2124
report (one sorted line per command, outcome tag only) that CI byte-compares
2225
to detect coverage drift. The two are mutually exclusive.
2326
24-
Exit code: nonzero iff any `Fail`, `InterpreterError`, or `OutOfFuel`
25-
outcome was recorded. `DecodeError`/`ModuleUnavailable`/`Skipped` don't
26-
fail the run — they're "feature not implemented" signal, not regressions.
27-
`--report` always exits 0 (the report diff, not the exit code, is the gate).
27+
Exit code: `1` iff any `Fail`, `InterpreterError`, or `OutOfFuel` outcome was
28+
recorded, and `3` for a setup failure (bad arguments, an unreadable testsuite
29+
directory, no matching files, or temp-dir creation failure). `DecodeError`/
30+
`ModuleUnavailable`/`Skipped` don't fail the run — they're "feature not
31+
implemented" signal, not regressions. `--report` always exits 0 (the report
32+
diff, not the exit code, is the gate).
2833
-/
2934

3035
namespace Wasm.Testsuite

0 commit comments

Comments
 (0)