docs: separate benchmark material from product docs (#2019)

* docs: separate benchmark material from product docs

* docs: keep product docs focused

* docs: add benchmark documentation index

* docs: remove frozen admission audit history

* tests: drop removed audit exemption

* docs: retarget evaluation methods to benchmark docs

* docs: keep internalcot note with benchmark evidence

* docs: move Harbor authoring guide under benchmarks

* docs: move agent evaluation guide under benchmarks

* docs: move MCP visibility guide under benchmarks

* docs: tighten benchmark contracts

* docs: point benchmark README at benchmark docs

* docs: update benchmark template links

* tooling: validate benchmark docs at new paths

* docs: point Harbor skill at benchmark docs

Author: morluto

.agents/skills/harbor-benchmarks/SKILL.md

@@ -147,8 +147,7 @@ submitted claim.
 ## Validate
 
 Read `AGENTS.md`, `CONTRIBUTING.md`, and
-`docs/reference/evaluations/benchmark-contracts.md`. Use the pinned runner and
-planner:
+`benchmarks/docs/benchmark-contracts.md`. Use the pinned runner and planner:
 
 ```sh
 uvx --from harbor==0.20.0 harbor --version

benchmarks/README.md

@@ -6,6 +6,9 @@ which mathematical subject the task belongs to. Subject organization lives in
 the required `primary_domain` taxonomy and the more detailed `field` metadata;
 tasks remain direct children of their dataset roots.
 
+See [benchmark documentation](docs/index.md) for task authoring, evaluation
+methods, and operator-run observations.
+
 `benchmarks/datasets/<dataset>/` is the Harbor dataset root and contains the
 dataset's executable task bundles directly. `members/` retains Jacobian's
 authoritative identity, provenance, assurance, provider, environment-profile,
@@ -83,7 +86,7 @@ make codex-visibility
 mathematical correctness task. It compares no Jacobian with Jacobian MCP only
 and measures whether Codex discovers, inspects, invokes, completes, or
 independently checks relevant operations. See
-[Run the MCP visibility evaluation](../docs/how-to/run-codex-visibility-evaluation.md).
+[Run the MCP visibility evaluation](docs/run-codex-visibility-evaluation.md).
 
 `symbolic-coordination-v1` keeps its deterministic 26-case pilot separate
 from the fixed `mathematical-benchmarks-v1` snapshots. Its task bundles are solvable
@@ -161,8 +164,8 @@ Performance timing is reported separately from reward, and research datasets
 are explicitly non-comparative diagnostics. Uniform task structure does not
 make rewards across these datasets comparable.
 
-See [authoring a Harbor benchmark task](../docs/how-to/author-harbor-benchmark-task.md),
-[benchmark contracts](../docs/reference/evaluations/benchmark-contracts.md), and the
+See [authoring a Harbor benchmark task](docs/author-harbor-benchmark-task.md),
+[benchmark contracts](docs/benchmark-contracts.md), and the
 [Harbor benchmarks skill](../.agents/skills/harbor-benchmarks/SKILL.md).
 
 ## Research loop and subject taxonomy

…s/how-to/author-harbor-benchmark-task.md …rks/docs/author-harbor-benchmark-task.mddocs/how-to/author-harbor-benchmark-task.md renamed to benchmarks/docs/author-harbor-benchmark-task.md docs/how-to/author-harbor-benchmark-task.md renamed to benchmarks/docs/author-harbor-benchmark-task.md

@@ -1,18 +1,18 @@
 # Author a Harbor benchmark task
 
-[Documentation home](../index.md) · [Benchmark contracts](../reference/evaluations/benchmark-contracts.md)
+[Benchmark home](../README.md) · [Benchmark contracts](benchmark-contracts.md)
 
 Harbor tasks are external evaluation assets. Choose a bounded, difficult
 mathematical claim because it reveals a capability needed for serious
 mathematical investigation, not because an existing Jacobian operation makes
 it easy. A task can expose an operation gap; tool availability belongs in the
 experimental treatment, not in the definition of a good task.
 
-Copy [the task template](../../benchmarks/templates/task/README.md) into a
-registered dataset. Keep `instruction.md` and `environment/` agent-visible;
-keep `solution/` and `tests/` Oracle/verifier-only. Do not copy an existing
-task's ceremonial `answer.txt`, hidden `expected.json` predicate, keyword
-gate, or universal certificate union into a new task.
+Copy [the task template](../templates/task/README.md) into a registered dataset.
+Keep `instruction.md` and `environment/` agent-visible; keep `solution/` and
+`tests/` Oracle/verifier-only. Do not copy an existing task's ceremonial
+`answer.txt`, hidden `expected.json` predicate, keyword gate, or universal
+certificate union into a new task.
 
 ## Write the public contract first
 

benchmarks/docs/benchmark-contracts.md

@@ -0,0 +1,75 @@
+# Benchmark contracts
+
+[Benchmark home](../README.md)
+
+Harbor benchmarks are external evaluation assets. A task owns its
+agent-visible instruction, hidden Oracle material, verifier, pinned environment,
+and task digest under `benchmarks/datasets/`. Jacobian is an experimental
+treatment the benchmark may expose; benchmark design does not extend Jacobian's
+public API.
+
+Choose tasks for the mathematical capability they measure, including
+capabilities the current operation library cannot solve. Freeze task and
+environment identity, keep hidden material out of the agent environment, and
+report limitations with the result.
+
+## Public submission contract
+
+For an atomic mathematical task, reward is normally binary: `1` only when the
+verifier can replay the submitted result against the frozen input and every
+declared witness condition holds; otherwise `0`. Tool calls, prose, confidence,
+and diagnostic observations do not earn credit.
+
+The public submission is normally `{ "result": ... }`. Add `"witness"` only
+for a finite task-specific mathematical object that replay cannot derive from
+the input and result. Do not require duplicate result files, prose
+explanations, leaked conclusions, or verifier-derived status fields.
+
+The instruction, submission schema, hidden solver, and verifier must accept the
+same objects. When a schema changes, update all four together plus the gold
+submission and public-contract fixture.
+
+## Replay authority
+
+Correctness comes from task-local replay using the frozen verifier copy of
+`input.json` and the submitted mathematical value. `tests/expected.json` is an
+Oracle regression fixture, not the definition of correctness. Changing only an
+expected fixture must not change reward for a fixed input/submission pair.
+
+Independent claims must be checked independently. A corrupted field or failed
+subclaim must not decide an unrelated collision, invertibility, completeness,
+or witness claim.
+
+## Mathematical representations
+
+Compare represented mathematical values rather than preferred renderings:
+
+- parse structured rationals as exact fractions;
+- treat sets, maps, distributions, and sparse polynomials as unordered unless
+  order is mathematically part of the task;
+- represent formulas with the smallest task-owned structure rather than scored
+  prose;
+- require canonical form only when canonicalization is itself a stated outcome.
+
+Reject malformed types, booleans used as integers, invalid denominators,
+non-finite values, and resource-bound violations at the boundary.
+
+## Witnesses and artifacts
+
+`answer.txt` is not an authoritative submission interface. A witness artifact
+is justified only when replay needs an external finite object; it must not
+mirror or hash `result`, carry boilerplate prose, or exist merely because an
+older task used one. Gold witness paths must resolve to regular files beneath
+the task's `solution/` root.
+
+Generated task families should expose only the certificate variants their
+public claim can reward. Do not publish one universal certificate union across
+unrelated claim families.
+
+## Evaluation output
+
+Task bundles and immutable snapshot locks are reproducibility inputs. Run
+outputs, trajectories, reports, and other regenerable evidence belong under
+ignored `benchmarks/results/` or external artifact storage. A benchmark result
+is evidence about the experiment; it is not a new mathematical or product
+contract for Jacobian.

…erence/evaluations/evaluation-methods.md benchmarks/docs/evaluation-methods.mddocs/reference/evaluations/evaluation-methods.md renamed to benchmarks/docs/evaluation-methods.md docs/reference/evaluations/evaluation-methods.md renamed to benchmarks/docs/evaluation-methods.md

@@ -1,6 +1,6 @@
 # Evaluation methods
 
-[Documentation home](../../index.md)
+[Benchmark home](../README.md) · [Benchmark contracts](benchmark-contracts.md)
 
 Jacobian evaluations are operator-run evidence exercises, not server features
 or routine pull-request gates. Compare a control with no Jacobian against a
@@ -14,12 +14,11 @@ then asks whether the current tool surface helps on that fixed capability set;
 persistent failures are evidence for a future operation or environment change.
 
 Each Harbor task owns its hidden verifier and Oracle. Author that contract from
-the [task template](../../../benchmarks/templates/task/README.md) and
+the [task template](../templates/task/README.md) and
 [benchmark contracts](benchmark-contracts.md); do not copy an existing task's
 hidden `expected.json` predicate, lowest-terms wording, keyword gate, or
-universal certificate union. Report mathematical
-correctness, tool use, failure modes, cost, and the limits of the task set
-separately. Atomic mathematical task correctness is normally binary; only
-explicit independent replayable subclaims justify partial credit. An evaluation
-score or solver outcome is evidence about the experiment, not a new
-mathematical conclusion returned by Jacobian.
+universal certificate union. Report mathematical correctness, tool use, failure
+modes, cost, and the limits of the task set separately. Atomic mathematical task
+correctness is normally binary; only explicit independent replayable subclaims
+justify partial credit. An evaluation score or solver outcome is evidence about
+the experiment, not a new mathematical conclusion returned by Jacobian.

benchmarks/docs/index.md

@@ -0,0 +1,14 @@
+# Benchmark documentation
+
+[Benchmark home](../README.md)
+
+These documents describe Harbor datasets and operator-run evaluations. They are
+research and benchmark infrastructure, not part of Jacobian's public server
+contract.
+
+- [Author a Harbor benchmark task](author-harbor-benchmark-task.md)
+- [Benchmark contracts](benchmark-contracts.md)
+- [Evaluation methods](evaluation-methods.md)
+- [Run agent observations](run-agent-evaluations.md)
+- [Run the MCP visibility evaluation](run-codex-visibility-evaluation.md)
+- [Internalcot visible-reasoning intervention](internalcot-visible-reasoning-intervention.md)

benchmarks/docs/internalcot-visible-reasoning-intervention.md

@@ -0,0 +1,10 @@
+# Internalcot visible-reasoning intervention
+
+[Benchmark home](../README.md)
+
+Status: **historical, research-only**.
+
+This records an external paired evaluation of an Internalcot intervention. It
+is benchmark evidence, not part of Jacobian's server contract. Any product
+change motivated by the study requires a separate current architecture decision
+and fresh evaluation.

docs/how-to/run-agent-evaluations.md benchmarks/docs/run-agent-evaluations.mddocs/how-to/run-agent-evaluations.md renamed to benchmarks/docs/run-agent-evaluations.md

@@ -1,6 +1,6 @@
 # Run agent observations
 
-[Documentation home](../index.md) · [Evaluation reference](../reference/evaluations/evaluation-methods.md)
+[Benchmark home](../README.md) · [Evaluation reference](evaluation-methods.md)
 
 Use Harbor to see how an agent performs on a bounded mathematical task with the
 public Jacobian tools available. A normal run is an observation: Harbor records

…ow-to/run-codex-visibility-evaluation.md …/docs/run-codex-visibility-evaluation.mddocs/how-to/run-codex-visibility-evaluation.md renamed to benchmarks/docs/run-codex-visibility-evaluation.md docs/how-to/run-codex-visibility-evaluation.md renamed to benchmarks/docs/run-codex-visibility-evaluation.md

@@ -1,11 +1,11 @@
 # Run the MCP visibility evaluation
 
+[Benchmark home](../README.md)
+
 This external diagnostic compares a no-Jacobian control with a treatment that
 exposes only Jacobian's public MCP server. It measures discovery and use of
 `math.find` and `math.run`; it does not replace mathematical task evaluation.
 
 Freeze the prompt suite, model, environment, and endpoint configuration before
 running either arm. Store the raw outputs with the evaluation's own metadata
-and interpret them as research evidence only. The diagnostic must not add a
-reasoning observer, retained workflow state, or new MCP operation to
-Jacobian.
+and interpret them as research evidence only.

benchmarks/templates/task/README.md

@@ -31,5 +31,5 @@ a witness path to `evidence/answer.txt`. Do not copy a universal certificate uni
 into a generated family; each task's schema admits only the certificate that
 family can reward. Instruction, schema, solver, and verifier must accept
 the same objects. Closed success variants omit inapplicable fields. See
-[authoring a Harbor benchmark task](../../../docs/how-to/author-harbor-benchmark-task.md)
-and [benchmark contracts](../../../docs/reference/evaluations/benchmark-contracts.md).
+[authoring a Harbor benchmark task](../../docs/author-harbor-benchmark-task.md)
+and [benchmark contracts](../../docs/benchmark-contracts.md).