BaseVerifier.verify() takes only timeout_sec, and every built-in verifier reads cluster state — so a verifier that checks the agent's final response text or tool trace has nothing to read. The data exists: AgentResult.output and AgentResult.trajectory are in scope at run_task's verification call site (evalharness/default.py: execute_agent at ~803, _run_verification at ~827), but _run_verification passes only (entry, timeout_sec) down to the leaves, and the result record is written after verification runs.
Downstream we (gke-labs/kube-agents) worked around it: our AgentHarness.run() stashes output+trajectory in module state, and our entry-point verifiers (report_contains, tool_called) read the stash. It works because task execution is single-threaded, but it has a real residual: on the exception path where run_task still verifies although the harness raised before execute_agent, the stash holds the previous task's transcript. Fail-closed guards contain it; threading the data properly removes it.
Proposal: pass agent_res into _run_verification and set it on verifiers that declare an opt-in field (e.g. a wants_transcript marker or a typed context attribute), keeping verify(timeout_sec) source-compatible for existing verifiers. Happy to send a PR if the shape sounds right.
Written with the help of Claude as part of kube-agents' testing work; pinned SHA in our deployment is 4670d76.
BaseVerifier.verify() takes only timeout_sec, and every built-in verifier reads cluster state — so a verifier that checks the agent's final response text or tool trace has nothing to read. The data exists: AgentResult.output and AgentResult.trajectory are in scope at run_task's verification call site (evalharness/default.py: execute_agent at ~803, _run_verification at ~827), but _run_verification passes only (entry, timeout_sec) down to the leaves, and the result record is written after verification runs.
Downstream we (gke-labs/kube-agents) worked around it: our AgentHarness.run() stashes output+trajectory in module state, and our entry-point verifiers (report_contains, tool_called) read the stash. It works because task execution is single-threaded, but it has a real residual: on the exception path where run_task still verifies although the harness raised before execute_agent, the stash holds the previous task's transcript. Fail-closed guards contain it; threading the data properly removes it.
Proposal: pass agent_res into _run_verification and set it on verifiers that declare an opt-in field (e.g. a wants_transcript marker or a typed context attribute), keeping verify(timeout_sec) source-compatible for existing verifiers. Happy to send a PR if the shape sounds right.
Written with the help of Claude as part of kube-agents' testing work; pinned SHA in our deployment is 4670d76.