Maps every release criterion of the 0.2.0 vertical slice to the concrete
evidence that proves it. Benchmark numbers below are the measured values from
BENCHMARK-RESULTS.md (generated by python benchmarks/run.py --write on
CPython 3.11.14, macOS arm64; thresholds re-enforced on every CI run on
Linux/macOS/Windows).
| # | Criterion | Evidence |
|---|---|---|
| 1 | Canonical versioned trace schema | TRACE-SPEC.md; schemas/trace.schema.json validated against every sample (tests/test_schemas.py); unknown majors rejected (tests/test_adversarial.py::test_unknown_schema_version_rejected) |
| 2 | Upgraded policy schema with precise validation errors | POLICY-SPEC.md; tests/test_argspec.py, tests/test_adversarial.py::TestMalformedPolicies |
| 3 | Deterministic trajectory rules | tests/test_trajectory_rules.py (30 tests); benchmark categories ordering/side_effects/error_handling/completion all detected: 34/34 cases, precision 1.0000, recall 1.0000, 0 FP, 0 FN |
| 4 | Baseline-vs-candidate regression comparison | tests/test_regression.py (14 tests), tests/test_cli_compare.py; benchmark regression scenarios: identical_run_passes=True, three seeded regressions fail the gate, fix_passes_gate=True |
| 5 | Real provider adapters with fixtures | tests/test_adapters.py (realistic OpenAI/Anthropic fixtures incl. error results, orphan tool results, unparseable arguments); benchmark adapter equivalence = True (same conversation, both providers, identical verdicts) |
| 6 | JSON, Markdown, JUnit, SARIF reports | tests/test_report.py, tests/test_report_formats.py (XML parses under hostile characters; SARIF 2.1.0 structure, rule index consistency, severity mapping) |
| 7 | Deterministic reports across repeated runs | benchmark determinism: byte_identical=True; tests/test_cli_compare.py::TestDeterministicOutputs (all four formats byte-compared via CLI) |
| 8 | GitHub Action integration | action.yml (composite, no secrets); the same command paths smoke-tested in .github/workflows/ci.yml on ubuntu/macos/windows |
| 9 | Realistic benchmark fixtures + methodology | benchmarks/cases/ (34 seeded cases), benchmarks/tools/generate_cases.py (deterministic; CI checks cases are in sync), BENCHMARK-METHODOLOGY.md (incl. explicit out-of-scope list) |
| 10 | Backward compatibility / migration path | tests/test_compat_golden.py (v0.1 sample verdicts and messages pinned; scores 100.0/80.0 unchanged, unsafe sample 70.0->70.83 due to the documented nested-safety-scan fix); legacy baselines match in compare (TestLegacyBaselineCompatibility); COMPATIBILITY.md |
| 11 | Reproducible detection of seeded failures | benchmark detection: 34/34 cases correct, precision 1.0, recall 1.0 (exact TP count: see BENCHMARK-RESULTS.md, the generated single source of truth) |
| 12 | Documented precision/recall | BENCHMARK-RESULTS.md (precision 1.0000, recall 1.0000) with the seeded-scope caveat in BENCHMARK-METHODOLOGY.md |
| 13 | Performance | measured numbers in BENCHMARK-RESULTS.md (generated; threshold < 50 ms/trace, CI-enforced) |
| 14 | Provider-neutral core | no adapter imports in core modules (ARCHITECTURE.md isolation rule); adapter-equivalence benchmark = True |
| 15 | Clean installation in a fresh environment | CI package job: python -m build, wheel installed into a fresh venv, arh --version + sample validation smoke |
| 16 | Useful first result in under five minutes | docs/quickstart.md (every command executed by tests/test_docs_examples.py); zero runtime dependencies |
| 17 | Malformed/adversarial input handling | tests/test_adversarial.py (truncated/empty/wrong-type JSON, invalid status, negative tokens, invalid regex, 5000-step trace, 200-deep nesting, hostile unicode in every renderer, injection-as-data) |
| 18 | Metamorphic robustness | benchmark metamorphic invariances: 102 checks, 0 failures (trace-id rename, metadata noise, policy key-order) |
| 19 | Type/lint gates | mypy --strict on src (0 issues), ruff check clean — both CI jobs |
| 20 | No unresolved critical/high verifier findings | Phase-9 independent reviews; resolution recorded in the release notes / final report |
Test totals at spec time: 165 unit/CLI/adapter/schema tests plus the
benchmark runner's threshold checks. Full suite:
python -m unittest discover -s tests and python benchmarks/run.py.