feat(benchmarks): add selected-task workflow (#572)

Author: morluto

.agents/skills/harbor-benchmarks/SKILL.md

@@ -111,17 +111,26 @@ Use the pinned Harbor runner from the repository:
 ```sh
 uvx --from harbor==0.20.0 harbor --version
 make harbor-plan BASE=origin/main
-make harbor-sync
-make harbor-check-task DATASET=mathematical-benchmarks-v1 TASKS="task-id"
-make harbor-oracle-task DATASET=mathematical-benchmarks-v1 TASKS="task-id"
+make harbor-prepare-task DATASET=mathematical-benchmarks-v1 TASKS="task-id"
+make harbor-validate-task DATASET=mathematical-benchmarks-v1 TASKS="task-id"
 ```
 
-The selected-task commands are the normal leaf-task gates and require an
-explicit task selection. Use the full `make harbor-check` and explicitly
+`harbor-prepare-task` is the explicitly mutating authoring step. It formats only
+the selected task Python and dedicated validation leaf, performs scoped public
+contract and verifier checksum synchronization, and reports exactly which
+generated files changed. `harbor-validate-task` is the complete source-read-only
+leaf gate: it resolves membership and planner-owned host selectors once, runs
+static quality before contracts, executes the selected leaf and generic tests
+with an isolated worktree-local pytest directory, then runs exact task Oracles
+serially and reports timings, digests, and evidence paths.
+
+Both commands require an explicit task selection. The lower-level
+`harbor-sync`, `harbor-check-task`, and `harbor-oracle-task` targets remain
+available for a narrow edit loop. Use the full `make harbor-check` and explicitly
 scoped `make harbor-oracle` paths for shared tooling, schemas, registry, suite
-policy, or other control-plane changes. A full dataset Oracle requires
-`FULL=1`; ordinary Oracle runs require `TASKS` and never expand an omitted
-selection implicitly.
+policy, or other control-plane changes. A full dataset Oracle requires `FULL=1`;
+ordinary Oracle runs require `TASKS` and never expand an omitted selection
+implicitly.
 
 After any input, instruction, metadata, verifier, dependency, image, or task
 contract change:
@@ -157,9 +166,11 @@ formatting applies to the entire `benchmarks/` tree, including task verifier
 code under `benchmarks/datasets/<dataset>/<task-id>/tests/verifier.py`.
 
 After any verifier Python (`tests/verifier.py`) or Dockerfile change, run
-`make harbor-sync` to update the verifier checksum label embedded in each
-task's `tests/Dockerfile`. The `harbor-contracts` gate rejects stale checksum
-labels; `harbor-sync` recomputes them from the current verifier source.
+`make harbor-prepare-task DATASET=... TASKS="..."` to update the verifier
+checksum label embedded in each task's `tests/Dockerfile`. The
+`harbor-contracts` gate rejects stale checksum labels; the preparation command
+uses the scoped `harbor-sync` operations to recompute them from current verifier
+source.
 
 ### Validation regression layout
 

.agents/skills/verifier-evaluations/SKILL.md

@@ -23,6 +23,15 @@ the Harbor skill owns dataset layout and repository commands.
    solution, hidden verifier logic, private authorization records, or Oracle
    fixtures merely to explain the format.
 
+   Use `tests/verifier_contract.json` as the sole task-local declaration of
+   behavior consumed by generic verifier tests. Keep it versioned and validate
+   it against a closed schema: boolean fields must be exact JSON booleans,
+   unknown keys and unsupported versions fail closed, and missing metadata must
+   not silently opt a task into exceptional behavior. Do not add parallel
+   metadata files, fields on unrelated public-contract models, or global
+   task-name registries for input-binding, scope, assurance, or diagnostic
+   exceptions.
+
 2. Trace every acceptance path:
 
    `input file → envelope → typed structure → semantic claim → evidence/scope → metrics → reward`
@@ -63,6 +72,15 @@ the Harbor skill owns dataset layout and repository commands.
    documented in the visible contract, reject unrelated text, and accept
    mathematically equivalent phrasing.
 
+   Minimize the semantic obligations before implementing prose checks. Require
+   only logically independent facts that the evidence must contribute. If the
+   typed certificate plus one checked fact already entails a conclusion, do
+   not also require a rhetorical sentence restating that conclusion. For
+   example, a verifier that proves the submitted corrected condition fails
+   must not additionally require the solver to say “therefore this does not
+   refute the repair.” Keep such implications in verifier-owned mathematics,
+   not in preferred wording.
+
    For streamed prose verifiers, preserve the local relationship between the
    claim, its scope, and any negation; independent lexical matches are not a
    sufficient semantic parser. Add regressions for scope-before-claim and
@@ -86,6 +104,13 @@ the Harbor skill owns dataset layout and repository commands.
    verifier still emits `reward.json` without crashing. Include a large valid
    evidence artifact to prove that no undocumented byte cap is present.
 
+   Include at least one terse numeric or structural explanation fixture built
+   independently from the public contract. It must express the required facts
+   without reusing the canonical answer's labels, rhetorical conclusions, or
+   sentence fragments. Do not inspect hidden solution text solely to construct
+   this fixture; the point is to prove semantic acceptance rather than encode a
+   second preferred answer.
+
 7. Validate the final tree and handoff. Run focused tests, deliberate negative
    cases, the selected Oracle, and the repository's planned gate. If shared
    verifier support changes, migrate only deliberately selected task-local
@@ -97,6 +122,15 @@ the Harbor skill owns dataset layout and repository commands.
    it matches `sha256sum` of the final `verifier.py`; a stale label fails
    `validate_task_topology` and blocks the task.
 
+   If an Oracle run earns full mathematical correctness but zero evidence
+   validity, diagnose the prose recognizer without reading or copying hidden
+   answer text. Instrument or invoke the matcher to report only a boolean map
+   of documented semantic clauses and contradiction checks. Do not print,
+   tokenize, quote, or mine n-grams from the Oracle artifact. Repair the public
+   semantic rule, write a fresh regression from the visible contract, and then
+   rerun every check invalidated by the verifier change, including the exact
+   Oracle.
+
 Read [references/verifier-contract.md](references/verifier-contract.md) for the
 detailed checklist and anti-pattern catalogue.
 

.github/scripts/plan-benchmarks

@@ -64,6 +64,7 @@ BENCHMARK_CONTROL_PATHS = frozenset(
         "tools/check_benchmark_contracts.py",
         "tools/check_benchmark_static.py",
         "tools/check_harbor_dataset.py",
+        "tools/harbor_task_workflow.py",
         "tools/sync_harbor_verifier_support.py",
     }
 )

CONTRIBUTING.md

@@ -21,7 +21,7 @@ capability membership.
 | Documentation, benchmark README, or `benchmarks/validation/` | `make docs-linkcheck` | None |
 | Focused Python behavior | `make test-plan BASE=<revision>`, the selected lane, then `make check` | None |
 | Harbor job JSON, MCP config, job-level Compose overlay, or execution helper | `make harbor-execution-check` | None |
-| Benchmark task input or verifier | `make harbor-check-task DATASET=... TASKS=...`, then `make harbor-oracle-task DATASET=... TASKS=...` | Exact selected-task Oracle |
+| Benchmark task input or verifier | `make harbor-prepare-task DATASET=... TASKS="..."`, then `make harbor-validate-task DATASET=... TASKS="..."` | Exact selected-task Oracle |
 | Deployment entrypoint | `make deploy-check` | None |
 | CI, dependencies, or unknown paths | `make check-static` plus affected tests | As required by the selected plan |
 
@@ -77,6 +77,18 @@ modules for focused changes, the changed validation file itself, and the full
 `benchmarks/validation` suite, sharded in hosted CI, for shared or unclassified
 infrastructure. To reproduce one selected host check locally, run
 `make harbor-validation-tests TESTS=<pytest-file-or-directory>`.
+
+For task authoring, `make harbor-prepare-task DATASET=... TASKS="..."` is the
+explicitly mutating preparation step. It formats only Python owned by the
+selected task and its dedicated validation leaf, runs scoped public-contract
+and verifier-checksum synchronization, and reports every generated file that
+changed. Follow it with `make harbor-validate-task DATASET=... TASKS="..."` for
+the complete source-read-only leaf gate. That command resolves membership and
+planner selectors once, fails fast through static quality and contracts, runs
+the selected host tests serially in a worktree-local temporary pytest directory,
+then runs each exact Oracle serially. Its summary includes per-stage timings and
+the resulting task digest and Oracle evidence path.
+
 Tests can be narrowed without learning another wrapper:
 
 ```sh
@@ -151,7 +163,7 @@ aid; `make check` and CI remain the handoff gates.
 | --- | --- | --- |
 | Docs only | `make docs-linkcheck` | Documentation |
 | Focused Python | affected target, then `make check` | Planned Python/static/package lanes |
-| Benchmark task or verifier | `make harbor-check-task DATASET=... TASKS=...` and `make harbor-oracle-task DATASET=... TASKS=...` | Exact task contract and Oracle |
+| Benchmark task or verifier | `make harbor-prepare-task DATASET=... TASKS="..."`, then `make harbor-validate-task DATASET=... TASKS="..."` | Exact task contract, planner-selected host tests, and Oracle |
 | Benchmark README or validation regression | focused Harbor checks | Contract checks; no Oracle |
 | Lean runtime | focused `make test-lean`, then `make check` | Lean plus affected lanes |
 | CI, dependencies, or unknown paths | `make check-static` plus affected tests | Fail-closed functional lanes |

Makefile

@@ -15,7 +15,7 @@ ORDERING_DEFAULT_SEED := --randomly-seed=17
 PYTEST_DIAGNOSTIC_ARGS ?= --durations=10
 RUFF_PATHS := src tests benchmarks
 TOPOLOGY_RUNNER := $(UV_RUN) python tools/test_topology.py
-PUBLIC_COMMANDS := help setup check check-changed ci-plan test-plan test-changed test-unit test-component test-domain test-composition test-storage test-process test-mcp test-provider test-lean test-e2e docs-command-check docs-linkcheck harbor-plan harbor-execution-check harbor-check-task harbor-oracle-task npm-test test-all-ci check-static deploy-check
+PUBLIC_COMMANDS := help setup check check-changed ci-plan test-plan test-changed test-unit test-component test-domain test-composition test-storage test-process test-mcp test-provider test-lean test-e2e docs-command-check docs-linkcheck harbor-plan harbor-prepare-task harbor-validate-task harbor-execution-check harbor-check-task harbor-oracle-task npm-test test-all-ci check-static deploy-check
 
 ifneq ($(strip $(PATHS)),)
 PATHS_FILE := $(shell mktemp)
@@ -26,7 +26,7 @@ endif
 # in pyproject.toml: direct pytest invocations must not silently inherit a
 # signal-based deadline that cannot interrupt a native solver.  Process and
 # provider lanes run risky work in killable children and set their own deadline.
-.PHONY: help help-all uv-version-check setup setup-agent container-image eval-image eval-image-pull eval-image-bind hooks fix lint complexity-check lint-full security-audit typecheck test-architecture architecture ci-plan test-plan test-changed check-changed test-unit test-component test-domain test-composition test-storage test-process test-mcp test-provider test-lean test-e2e test-affected test-all-ci test-compatibility test-stress test-ordering duplicate-code npm-test todo-check coverage build check precommit check-static harbor-plan harbor-sync harbor-contracts harbor-execution-check harbor-adapter-checks harbor-validation-tests harbor-validate harbor-check harbor-check-task benchmark-inventory benchmark-snapshot benchmark-snapshot-validate benchmark-publish harbor-oracle harbor-oracle-task harbor-oracle-run harbor-oracle-all harbor-adapter-check heldout-validate heldout-render heldout-smoke agent-eval agent-eval-validate agent-eval-compare codex-visibility provider-eval clean docs-command-check docs-linkcheck deploy-check
+.PHONY: help help-all uv-version-check setup setup-agent container-image eval-image eval-image-pull eval-image-bind hooks fix lint complexity-check lint-full security-audit typecheck test-architecture architecture ci-plan test-plan test-changed check-changed test-unit test-component test-domain test-composition test-storage test-process test-mcp test-provider test-lean test-e2e test-affected test-all-ci test-compatibility test-stress test-ordering duplicate-code npm-test todo-check coverage build check precommit check-static harbor-plan harbor-prepare-task harbor-validate-task harbor-sync harbor-contracts harbor-execution-check harbor-adapter-checks harbor-validation-tests harbor-validate harbor-check harbor-check-task benchmark-inventory benchmark-snapshot benchmark-snapshot-validate benchmark-publish harbor-oracle harbor-oracle-task harbor-oracle-run harbor-oracle-all harbor-adapter-check heldout-validate heldout-render heldout-smoke agent-eval agent-eval-validate agent-eval-compare codex-visibility provider-eval clean docs-command-check docs-linkcheck deploy-check
 
 help: ## Show available developer commands.
 	@awk -v public="$(PUBLIC_COMMANDS)" 'BEGIN {FS = ":.*## "; n = split(public, names, " "); for (i = 1; i <= n; i++) wanted[names[i]] = 1; printf "Jacobian common developer commands:\n\n"} /^[a-zA-Z_-]+:.*## / && ($$1 in wanted) {printf "  %-18s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
@@ -303,6 +303,18 @@ harbor-plan: ## Print the independent Harbor benchmark plan (BASE=... optional).
 	echo "Plan receipt:"; \
 	cat "$$tmp_dir/receipt.json"
 
+harbor-prepare-task: ## Format and sync selected Harbor tasks (DATASET=..., TASKS="...").
+	@test -n "$(DATASET)" || { echo "DATASET is required" >&2; exit 2; }
+	@test -n "$(TASKS)" || { echo "TASKS is required; refusing an unscoped preparation" >&2; exit 2; }
+	$(HARBOR_PYTHON) tools/harbor_task_workflow.py prepare \
+		--dataset "$(DATASET)" --tasks $(TASKS)
+
+harbor-validate-task: ## Run the complete selected-task static, host, and Oracle gate.
+	@test -n "$(DATASET)" || { echo "DATASET is required" >&2; exit 2; }
+	@test -n "$(TASKS)" || { echo "TASKS is required; refusing an implicit full-dataset validation" >&2; exit 2; }
+	$(HARBOR_PYTHON) tools/harbor_task_workflow.py validate \
+		--dataset "$(DATASET)" --tasks $(TASKS)
+
 harbor-sync: ## Update verifier checksum labels for selected tasks (DATASET=... TASKS="...").
 	@test -n "$(DATASET)" || { echo "DATASET is required" >&2; exit 2; }
 	@test -n "$(TASKS)" || { echo "TASKS is required; refusing an unscoped checksum update" >&2; exit 2; }

benchmarks/validation/test_benchmark_planner.py

@@ -129,6 +129,7 @@ def test_planner_digest_binds_to_planner_and_path_policy_sources() -> None:
         ".github/scripts/validate-benchmark-plan",
         ".github/workflows/benchmarks.yml",
         "Makefile",
+        "tools/harbor_task_workflow.py",
     ],
 )
 def test_benchmark_control_plane_changes_run_contract_checks(path: str) -> None: