Skip to content

Commit 3b37422

Browse files
committed
docs(skill): tighten Harbor evaluation boundaries
1 parent 6aea012 commit 3b37422

1 file changed

Lines changed: 18 additions & 0 deletions

File tree

  • .agents/skills/harbor-benchmarks

.agents/skills/harbor-benchmarks/SKILL.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,15 @@ configuration and MCP sidecar. Keep credentials, raw caches, host paths,
104104
floating dependencies, and Oracle/verifier material out of agent-visible
105105
files.
106106

107+
Validate that the agent image and runtime expose no solution, verifier, Oracle,
108+
repository-internal, or host-path material. Harbor's phase ordering and
109+
separate-verifier mode support this boundary but do not prove it.
110+
111+
Check that verifier timeout, network, CPU, memory, storage, and dependency
112+
behavior agree with the published task contract. Harbor applies configured
113+
limits subject to provider capabilities; Jacobian owns contract consistency and
114+
validation.
115+
107116
Before choosing the mutating workflow, confirm that the dataset uses the
108117
current public-contract shape. Some older or provider-focused bundles have a
109118
Harbor `task.toml` but a task-local `tests/public_contract.json` that is not
@@ -302,6 +311,10 @@ verifier status. Use Harbor's artifact manifest as the source of truth for
302311
artifact identity and reject traversal, escaping symlinks, missing entries, and
303312
non-conclusion execution states.
304313

314+
Bind committed studies to a retained immutable source revision or Jacobian
315+
snapshot identity. Reject relative or moving Git references such as `HEAD^`,
316+
branches, and unretained commits.
317+
305318
Current separate-verifier tasks retain task-local `tests/verifier_support.py`
306319
copies because Harbor requires the separate verifier image to contain its test
307320
runtime in the task `tests/` build context. The local copy is authoritative and
@@ -356,6 +369,11 @@ record flow, repeated or irrelevant calls, shell/file activity, tokens, time,
356369
cost, and completion. Record the git tree, task digests, provider/runtime,
357370
model/settings, prompt, seeds, raw traces, and structured reports.
358371

372+
For Jacobian studies that require reasoning telemetry, classify a trial as
373+
study-complete only after validating the required terminal reasoning event.
374+
Otherwise record it as `INCONCLUSIVE`. This classification is separate from
375+
Harbor's process and verifier completion.
376+
359377
## Handoff and publication
360378

361379
Report whether the result is task validation, a public regression, workflow

0 commit comments

Comments
 (0)