fix(benchmarks): harden Jacobian evaluation evidence - #946
Conversation
|
LGTM! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6187f3c4c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Do not call Jacobian for definitions, formatting, or non-execution tasks. | ||
|
|
||
| Call `math.run` directly for these stable contracts, preserving JSON types: | ||
| No discovery for stable producers: `{"capability_id":"<id>","mode":"EXPLORE","payload":<JSON>}` |
There was a problem hiding this comment.
Preserve optional discovery for stable capabilities
Replace the categorical “No discovery” instruction in both managed skill copies with an optional direct-call shortcut. This prompt now dictates the agent’s routing policy and biases visibility evaluations toward a preferred tool sequence; it can also prevent contract inspection when an installed catalog or capability version differs. Repository policy requires discovery and direct invocation to remain freely composable rather than prescribing their order.
AGENTS.md reference: AGENTS.md:L40-L46
Useful? React with 👍 / 👎.
Problem
The evaluation audit found two agent-hidden task requirements, repeat Oracle runs that could reuse prior Harbor jobs, old Compose/BuildKit cross-task Dockerfile reuse, and an avoidable discovery/token penalty for a stable combinatorics capability.
Changes
CODEX_HOMEcontrol/treatment boundaryTask versions 013 and 015 advance to 1.1.0 and 2.1.0. Oracle result directory names are now attempt-specific, and old Docker engines resolve to the evidence-bound legacy builder mode.
Validation
make harbor-host-validation: 3,601 passed across four shardsmake check: Ruff, formatting, complexity, mypy, 854 unit tests passedmake npm-test: 59 passed and package dry-run succeededmake docs-linkcheckmath.runcalls, 0 discovery calls, and 0 tool/parameter errors