Skip to content

Split verifier support and hard-gate evidence rewards - #615

Merged
morluto merged 8 commits into
mainfrom
codex/verifier-integrity-538-500
Aug 7, 2026
Merged

Split verifier support and hard-gate evidence rewards#615
morluto merged 8 commits into
mainfrom
codex/verifier-integrity-538-500

Conversation

@morluto

@morluto morluto commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Problem

The shared verifier validation modules had become large mixed-ownership hotspots, and mathematical benchmark rewards could still encode evidence as a soft additive term. That representation obscured the fail-closed invariant and made future verifier-policy migrations easier to get wrong.

Closes #500.
Closes #538.

Solution

Split the mathematical-benchmark and public-reproduction validation support into behavior-owned modules for fixtures, metadata, paths, verifier execution, evidence binding, input binding, assurance, and submission attacks.

Migrate the affected benchmark verifiers to hard-gate invalid evidence before reward aggregation. Remove redundant evidence terms from four remaining counterexample verifiers, strengthen the reward ratchet to recognize the ev alias, and synchronize the affected verifier checksum labels.

Testing

uv run --locked python -m pytest benchmarks/validation/test_fail_closed_reward_ratchet.py
uv run --locked python -m pytest \
  benchmarks/validation/mathematical_benchmarks_v1/test_subspace_direct_sum_counterexample.py \
  benchmarks/validation/mathematical_benchmarks_v1/test_polynomial_tail_counterexample.py \
  benchmarks/validation/mathematical_benchmarks_v1/test_matrix_square_zero_counterexample.py \
  benchmarks/validation/mathematical_benchmarks_v1/test_log_exponent_recovery.py

The ratchet reported 2 passing tests and the four affected verifier modules reported 5 passing tests. Earlier focused validation of the support split reported 1,699 mathematical-benchmark checks and 11 public-reproduction checks passing.

Trust & Compatibility Impact

This changes evaluation integrity behavior: invalid evidence now contributes zero reward rather than being represented inside a weighted aggregate. It does not authorize checkers or promote evaluator output to verified evidence. The support split preserves the existing public validation facade.

Checklist

  • Routine local validation passes (make check)
  • Relevant focused tests are listed above

@morluto
morluto marked this pull request as ready for review August 7, 2026 06:32
@cursor

cursor Bot commented Aug 7, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

cursoragent and others added 2 commits August 7, 2026 06:44
load_submission was rejecting schema-invalid submissions before scoring,
which collapsed correctness and hid false VERIFIED claims. Parse the
submission object for diagnostics, leave schema checks to the contract
helpers, and align adapter/legacy public-contract host validation.

Co-authored-by: morluto <morluto@users.noreply.github.com>
Evidence is now a hard reward gate. Update host validations that still
expected soft evidence bonuses, and keep protocol/evidence failures from
collapsing independent mathematical correctness.

Co-authored-by: morluto <morluto@users.noreply.github.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa6a565754

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

morluto added 3 commits August 7, 2026 15:26
# Conflicts:
#	benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-015/tests/Dockerfile
#	benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-016/tests/Dockerfile
@morluto
morluto merged commit 6536175 into main Aug 7, 2026
35 checks passed
@morluto
morluto deleted the codex/verifier-integrity-538-500 branch August 7, 2026 09:01
cursor Bot pushed a commit that referenced this pull request Aug 7, 2026
#615 landed load_submission_raw and reverted weakening load_submission,
which had collapsed host-verifier expectations (correctness staying 1.0
on envelope-only failures). Resolve conflicts by adopting main's tree so
this branch matches the supported diagnostic split.

Co-authored-by: morluto <morluto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants