test(verdict): golden-set regression harness for verdict prompt changes

[kyle-compute]

The gap

#1120 rewrote the verdict prompt (oddish/src/oddish/analyze/verdict_prompt.txt) and changed TaskVerdictModel from is_good: bool to verdict: Literal["accept","reject"]. The verdict is the judgment that decides whether a benchmark task is accepted, and nothing measured whether that rewrite (or any future one) shifts the accept/reject distribution. This PR adds that feedback loop as a golden set of seven scenarios, each pinning one rule of the prompt to the verdict it must produce:

case	prompt rule	expected
all_trials_good	accept when trials agree the task is fair	accept / high
majority_same_failure	reject when most trials show the same task problem	reject / high
single_bad_label	a single BAD label is normal for a difficult task	accept / low
all_harness_errors	harness errors say nothing about the task	accept / low
baseline_nop_passed	a failed baseline is an immediate rejection	reject / high
weak_tests_single_trial	weak tests reject; single-trial claim without a matching finding is medium	reject / medium
weak_tests_with_verifier_finding	same claim confirmed by a must_fix verifier finding is high	reject / high

Layer 1: offline (every CI run, no network)

tests/test_verdict_golden_set.py, over the shared fixtures in tests/verdict_golden_set.py:

• Rendering — VerdictBlock.build_prompt() succeeds for every scenario (a degraded section raises rather than rendering, so reaching the assertions proves it rendered) and the scenario's substance survives into the prompt: every trial's name and evidence, plus per-case markers like the CRITICAL: nop agent passed baseline line and the [must_fix/verifier] finding.
• Parsing — a synthetic judge reply for each case round-trips VerdictBlock.to_verdict() and reconstructs as TaskVerdictModel, guarding the verdict/confidence literals and the is_good property.
• A representativeness check keeps the set covering both verdicts and all three confidence levels.

This layer guards prompt-rendering and schema regressions. It says nothing about model behavior — that is deliberate and is what the second layer is for.

Layer 2: opt-in live backtest

tests/test_verdict_golden_set_live.py runs the REAL verdict model over the whole set through the production path (synthesize_task_verdict, including its fallback plumbing), prints a per-case table (case, expected, got, confidence) plus the aggregate accept rate, and fails if any case's verdict drifts. One case erroring records an error(...) row instead of hiding the rest of the table.

uv run pytest tests/test_verdict_golden_set_live.py -s

• Gating: skipped unless the credentials for settings.verdict_model's configured OpenAI provider resolve — the same require_public_openai_config() / require_azure_openai_config() checks _build_openai_client performs — so normal CI never touches the network.
• Comparing prompt versions: set ODDISH_VERDICT_PROMPT_FILE=/path/to/alt_prompt.txt and re-run. The test monkeypatches classifier._VERDICT_PROMPT (the module global build_verdict_prompt formats), so no production code changed; the override file must keep the four {num_trials}/{baseline_summary}/{quality_check_summary}/{trial_classifications} placeholders.
• Block persistence (save_to_s3 / save_to_db / record_cost) is patched out, matching the existing verdict-test style: the backtest measures the model, not S3/Postgres.

Testing

• uv run pytest tests/test_verdict_block.py tests/test_verdict_fallback.py tests/test_verdict_sync.py tests/test_verdict_golden_set.py tests/test_verdict_golden_set_live.py → 37 passed, 1 skipped (the live backtest, no credentials — confirming the gate).
• Smoke-tested the live path with a throwaway invalid key: the gate flips on, all seven cases run through synthesize_task_verdict, the table and accept rate print, and the final assertion reports the drifted cases.
• Verified the prompt-override seam changes build_verdict_prompt's output.

https://claude.ai/code/session_01MzJSdrHKEqJrJHU7RRJ758

---

Note

Low Risk
Test-only additions with no production code changes; live tests are credential-gated and skipped in normal CI.

Overview
Adds a seven-scenario golden set (verdict_golden_set.py) that maps each case to an expected accept/reject verdict and confidence, so prompt or schema rewrites can be regression-tested.

Offline CI (test_verdict_golden_set.py): every case must render via VerdictBlock.build_prompt() (trial evidence and scenario markers in the prompt) and synthetic judge JSON must round-trip through to_verdict() / TaskVerdictModel, including is_good. A meta-test keeps both verdicts and all three confidence levels covered.

Opt-in live backtest (test_verdict_golden_set_live.py): runs synthesize_task_verdict on the real model when OpenAI/Azure credentials resolve; prints a comparison table and fails on drift. Supports ODDISH_VERDICT_PROMPT_FILE to patch classifier._VERDICT_PROMPT for A/B prompt comparison, with persistence mocked out.

^{Reviewed by Cursor Bugbot for commit ff2286f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 9, 2026 12:39am

[cursor]

Risk: low. Approved — Cursor Bugbot passed with no findings requiring human review; Cursor Security Agent was not present. No reviewers assigned.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}

[github-actions]

Oddish preview

Commit: ff2286f96fef359547a0a29afb1cf59aebffeb4b

Surface	Link	Target
Frontend	https://pr-1135.oddish.app	Vercel preview for ff2286f
Backend	oddish-pr-1135	oddish-pr-1135
Database	project vcwffwrhjcrzshgdfcoe	project vcwffwrhjcrzshgdfcoe

Vercel deployment URL: https://oddish-c9ar48q77.oddish.app

Plan:

• Frontend deploy: true
• Backend deploy: true
• Migrations: false

This comment is updated by the PR Preview workflow.