Skip to content

fix(benchmarks): harden Jacobian evaluation evidence - #946

Merged
N0zoM1z0 merged 1 commit into
mainfrom
experiment/jcb-report-hardening
Aug 9, 2026
Merged

fix(benchmarks): harden Jacobian evaluation evidence#946
N0zoM1z0 merged 1 commit into
mainfrom
experiment/jcb-report-hardening

Conversation

@N0zoM1z0

@N0zoM1z0 N0zoM1z0 commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

The evaluation audit found two agent-hidden task requirements, repeat Oracle runs that could reuse prior Harbor jobs, old Compose/BuildKit cross-task Dockerfile reuse, and an avoidable discovery/token penalty for a stable combinatorics capability.

Changes

  • make the Problem 397 key facts and Problem 707 evidence schema fully agent-visible
  • assign every Oracle run a unique attempt while binding its stable experiment identity, task digests, job config, execution args, runtime versions, and result freshness into a sidecar receipt
  • serialize local Oracle builds and automatically use the classic builder on pre-23 Docker engines where Compose Bake can cross-load same-named Dockerfiles
  • add a bounded, installation-safe direct-run contract and latent visibility case for fixed-order difference-set extension
  • document the fresh CODEX_HOME control/treatment boundary

Task versions 013 and 015 advance to 1.1.0 and 2.1.0. Oracle result directory names are now attempt-specific, and old Docker engines resolve to the evidence-bound legacy builder mode.

Validation

  • make harbor-host-validation: 3,601 passed across four shards
  • make check: Ruff, formatting, complexity, mypy, 854 unit tests passed
  • focused visibility/Harbor unit lane: 55 passed
  • make npm-test: 59 passed and package dry-run succeeded
  • make docs-linkcheck
  • scoped 013/015 Harbor contract validation
  • exact 013 and 015 Oracle runs: 1/1 trial and reward 1.0 each
  • four Codex visibility experiments; final fixed-order case used 2 direct math.run calls, 0 discovery calls, and 0 tool/parameter errors

@N0zoM1z0

N0zoM1z0 commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

LGTM!

@N0zoM1z0
N0zoM1z0 merged commit 0052a5b into main Aug 9, 2026
28 checks passed
@N0zoM1z0
N0zoM1z0 deleted the experiment/jcb-report-hardening branch August 9, 2026 13:45

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6187f3c4c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Do not call Jacobian for definitions, formatting, or non-execution tasks.

Call `math.run` directly for these stable contracts, preserving JSON types:
No discovery for stable producers: `{"capability_id":"<id>","mode":"EXPLORE","payload":<JSON>}`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve optional discovery for stable capabilities

Replace the categorical “No discovery” instruction in both managed skill copies with an optional direct-call shortcut. This prompt now dictates the agent’s routing policy and biases visibility evaluations toward a preferred tool sequence; it can also prevent contract inspection when an installed catalog or capability version differs. Repository policy requires discovery and direct invocation to remain freely composable rather than prescribing their order.

AGENTS.md reference: AGENTS.md:L40-L46

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant