Skip to content

Commit dd0fd99

Browse files
committed
docs(evaluations): define reusable image contracts
1 parent 9bfca43 commit dd0fd99

3 files changed

Lines changed: 37 additions & 0 deletions

File tree

docs/explanation/prebuilt-agent-environment-direction.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ Jacobian tooling
2525
└─ records agent-image digest and MCP-server identity in observation evidence
2626
```
2727

28+
Formal-provider tasks use the same identity discipline but a different trust
29+
boundary: the agent image may carry an exploratory provider, while a separate
30+
digest-pinned checker image replays the submitted source artifact. A successful
31+
provider trace alone remains diagnostic evidence, not independent verification.
32+
The maintained Lean bases and their publication evidence are documented in
33+
[benchmark contracts](../reference/evaluations/benchmark-contracts.md#reusable-evaluation-images).
34+
2835
## Repository hooks (current)
2936

3037
- Task environments resolve through

docs/how-to/author-harbor-benchmark-task.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ task input and schemas; they do not run `apt-get`. Provider-specific tasks may
2626
install the dependencies that define that provider experiment. Use Harbor's
2727
native `public`, `no-network`, or `allowlist` task network policy. Proxy-backed
2828
operation belongs in the Harbor job composition and does not select an image.
29+
For a shared checker or provider runtime, keep the agent and verifier images
30+
separate and pin the published `@sha256:` reference; see
31+
[reusable evaluation images](../reference/evaluations/benchmark-contracts.md#reusable-evaluation-images).
2932

3033
Run `make harbor-plan BASE=origin/main`, inspect the prospective digest and
3134
exact changed-task lane, then validate and Oracle only the task being authored:

docs/reference/evaluations/benchmark-contracts.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,33 @@ digest-bound evidence, optional verification record, and limitations. Unknown
4848
fields fail closed. Task-specific schemas may narrow the result but cannot
4949
weaken the envelope.
5050

51+
### Reusable evaluation images
52+
53+
Use a shared image only when its toolchain is part of the task's reproducible
54+
runtime. The image digest, platform, toolchain version, and task digest together
55+
identify the executable evaluation; a tag is only a human-facing discovery
56+
label. Task Dockerfiles and environment profiles therefore use
57+
`name@sha256:<digest>`, never `main`, a version tag, or an unpinned base.
58+
59+
Keep agent/provider images separate from verifier images. An agent image may
60+
contain an exploratory service such as Lean REPL. A verifier image contains
61+
only the independently needed checker and its pinned dependencies, then
62+
replays a submitted artifact. Provider telemetry, tactic traces, and a
63+
successful agent-side process are useful diagnostics but do not authorize a
64+
mathematical conclusion.
65+
66+
The controlled `main`-only image workflow publishes the reusable Lean bases:
67+
68+
- `ghcr.io/morluto/jacobian-lean-checker` for Lean-source replay; and
69+
- `ghcr.io/morluto/jacobian-lean-repl-agent` for the pinned provider runtime.
70+
71+
Publication records the immutable image digest, source revision, platform,
72+
SBOM/provenance attestations, and unpacked image size. Measure runtime writable
73+
storage separately in the actual Harbor runner before setting `storage_mb`:
74+
image-layer size and writable-layer accounting are different quantities. Update
75+
an image pin only in a deliberate task-contract change, recompute prospective
76+
task digests, and rerun the selected Oracle.
77+
5178
## Task and verifier validation
5279

5380
Task and verifier validation is separate from model observation. For an

0 commit comments

Comments
 (0)