Skip to content

Commit d600db0

Browse files
yuelgrace1810-opsGrace Lee Rui Yuemorluto
authored
docs: reserve VERIFIED assurance in control evaluations (#906)
* docs: reserve verified assurance in control evaluations * docs: distinguish correctness from computed assurance * docs: bind evaluation assurance claims --------- Co-authored-by: Grace Lee Rui Yue <graceleeruiyue@mac.lan> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
1 parent 5ebc889 commit d600db0

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

docs/how-to/run-agent-evaluations.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,19 @@ make agent-eval DATASET=mathematical-benchmarks-v1 \
204204
MCP configuration. In the held-out runner, that condition records
205205
infrastructure status `NOT_CONFIGURED` and routing status `NOT_APPLICABLE`, and
206206
performs no Jacobian probe.
207+
208+
Keep the assurance vocabulary identical in both conditions. In particular,
209+
reserve `VERIFIED` for a result whose operator-authorized independent-checker
210+
record satisfies the task contract and binds the exact task input, claim,
211+
semantics, candidate, scope, certificate format, and checker identity. A correct
212+
manual derivation, self-written check, or source citation may support
213+
mathematical correctness, but does not itself establish `COMPUTED` and is not
214+
`VERIFIED`. A producer result carries only the assurance stated by its operation
215+
contract. Put this rule in any ad hoc prompt that does not already expose the
216+
task's structured submission contract. Otherwise a control answer can reuse
217+
`VERIFIED` as an ordinary English synonym and make the assurance comparison
218+
misleading.
219+
207220
`JACOBIAN_ENABLED=1` selects the treatment job and passes
208221
Harbor's `--mcp-config` option. `JACOBIAN_EVAL_PROXY=1` selects matching
209222
proxy-enabled control/treatment job configs and requires at least one proxy
@@ -275,6 +288,15 @@ Record the git tree, task digests, provider/runtime, model and prompt settings,
275288
raw trace location, and validation actually run. A public workflow result is
276289
regression or observation evidence; it is not held-out causal evidence.
277290

291+
For unstructured pilot answers, validate every claimed assurance level against
292+
its supporting evidence or operation contract and record every unsupported
293+
label as an assurance-vocabulary violation. In particular, `VERIFIED` requires
294+
the complete task-bound record above. Score assurance separately from
295+
mathematical correctness: a correct scalar or proof can still be correct while
296+
its assurance claim is false. Missing checker availability, a timeout, or
297+
failure to obtain a verification record is not evidence that the claim is
298+
false, but it also cannot raise the claim to `VERIFIED`.
299+
278300
## Troubleshooting
279301

280302
If the command exits with `JACOBIAN_MODEL must be exported`, export the model

0 commit comments

Comments
 (0)