Skip to content

docs: reserve VERIFIED assurance in control evaluations #906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morluto merged 3 commits into from
Aug 9, 2026
Merged

docs: reserve VERIFIED assurance in control evaluations #906

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
565a463
docs: reserve verified assurance in control evaluations
Aug 9, 2026
135713d
docs: distinguish correctness from computed assurance
morluto Aug 9, 2026
f0fe1b0
docs: bind evaluation assurance claims
morluto Aug 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/how-to/run-agent-evaluations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,19 @@ make agent-eval DATASET=mathematical-benchmarks-v1 \
MCP configuration. In the held-out runner, that condition records
infrastructure status `NOT_CONFIGURED` and routing status `NOT_APPLICABLE`, and
performs no Jacobian probe.

Keep the assurance vocabulary identical in both conditions. In particular,
reserve `VERIFIED` for a result whose operator-authorized independent-checker
record satisfies the task contract and binds the exact task input, claim,
semantics, candidate, scope, certificate format, and checker identity. A correct
manual derivation, self-written check, or source citation may support
mathematical correctness, but does not itself establish `COMPUTED` and is not
`VERIFIED`. A producer result carries only the assurance stated by its operation
contract. Put this rule in any ad hoc prompt that does not already expose the
task's structured submission contract. Otherwise a control answer can reuse
`VERIFIED` as an ordinary English synonym and make the assurance comparison
misleading.

`JACOBIAN_ENABLED=1` selects the treatment job and passes
Harbor's `--mcp-config` option. `JACOBIAN_EVAL_PROXY=1` selects matching
proxy-enabled control/treatment job configs and requires at least one proxy
Expand Down Expand Up @@ -275,6 +288,15 @@ Record the git tree, task digests, provider/runtime, model and prompt settings,
raw trace location, and validation actually run. A public workflow result is
regression or observation evidence; it is not held-out causal evidence.

For unstructured pilot answers, validate every claimed assurance level against
its supporting evidence or operation contract and record every unsupported
label as an assurance-vocabulary violation. In particular, `VERIFIED` requires
the complete task-bound record above. Score assurance separately from
mathematical correctness: a correct scalar or proof can still be correct while
its assurance claim is false. Missing checker availability, a timeout, or
failure to obtain a verification record is not evidence that the claim is
false, but it also cannot raise the claim to `VERIFIED`.

## Troubleshooting

If the command exits with `JACOBIAN_MODEL must be exported`, export the model
Expand Down
Loading