Merge remote-tracking branch 'github-main/main' into agent/discover-polynomial-power-normalization

Author: Grace Lee Rui Yue

.agents/skills/harbor-benchmarks/SKILL.md

@@ -104,6 +104,15 @@ configuration and MCP sidecar. Keep credentials, raw caches, host paths,
 floating dependencies, and Oracle/verifier material out of agent-visible
 files.
 
+Validate that the agent image and runtime expose no solution, verifier, Oracle,
+repository-internal, or host-path material. Harbor's phase ordering and
+separate-verifier mode support this boundary but do not prove it.
+
+Check that verifier timeout, network, CPU, memory, storage, and dependency
+behavior agree with the published task contract. Harbor applies configured
+limits subject to provider capabilities; Jacobian owns contract consistency and
+validation.
+
 Before choosing the mutating workflow, confirm that the dataset uses the
 current public-contract shape. Some older or provider-focused bundles have a
 Harbor `task.toml` but a task-local `tests/public_contract.json` that is not
@@ -302,6 +311,10 @@ verifier status. Use Harbor's artifact manifest as the source of truth for
 artifact identity and reject traversal, escaping symlinks, missing entries, and
 non-conclusion execution states.
 
+Bind committed studies to a retained immutable source revision or Jacobian
+snapshot identity. Reject relative or moving Git references such as `HEAD^`,
+branches, and unretained commits.
+
 Current separate-verifier tasks retain task-local `tests/verifier_support.py`
 copies because Harbor requires the separate verifier image to contain its test
 runtime in the task `tests/` build context. The local copy is authoritative and
@@ -356,6 +369,11 @@ record flow, repeated or irrelevant calls, shell/file activity, tokens, time,
 cost, and completion. Record the git tree, task digests, provider/runtime,
 model/settings, prompt, seeds, raw traces, and structured reports.
 
+For Jacobian studies that require reasoning telemetry, classify a trial as
+study-complete only after validating the required terminal reasoning event.
+Otherwise record it as `INCONCLUSIVE`. This classification is separate from
+Harbor's process and verifier completion.
+
 ## Handoff and publication
 
 Report whether the result is task validation, a public regression, workflow

.agents/skills/jacobian-math/SKILL.md

@@ -31,6 +31,8 @@ Call `math.run` directly for these stable contracts, preserving JSON types:
 - `polynomial.compute.gcd` in `EXPLORE` mode: payload keys are `left` and
   `right`; each value has shape
   `{"polynomial_schema_version":"1","domain":"QQ","variables":["x"],"polynomial":{"terms":[{"coefficient":{"num":"1","den":"1"},"exponents":[2]}]}}`.
+- For expression normalization, inspect the known
+  `polynomial.expression.normalize` contract directly.
 - `matrix.determinant.verify` in `VERIFY` mode for an independent check:
   `{"determinant_uri":"<determinant_uri from compute output>"}`.
 
@@ -56,15 +58,15 @@ artifact refs, including verification record URIs.
 Keep representation, decomposition, composition, iteration, verification
 timing, and stopping decisions agent-owned.
 
-When independent checking is requested, model-authored calculations or programs
-are not independent evidence. Use installed `VERIFY` when available. An artifact
-URI or checker summary is not a task-local verification-record file: never
-reconstruct or paraphrase such a record from returned fields. Claim `VERIFIED`
-only when the result has assurance level `VERIFIED`, exact record bytes, and
-required task authorization and bindings are preserved; otherwise use lower
-task-permitted assurance. Verification is bound to the exact checked claim: do
-not transfer `VERIFIED` from an input, premise, factorization, or related
-artifact to a model-derived conclusion, which needs its own checker-bound record.
+Model-authored calculations or programs are not independent evidence. Use
+installed `VERIFY` when requested; a writable path or schema alone is not
+authorization. For task-level `VERIFIED`, require result assurance `VERIFIED`,
+exact record bytes, and that required task authorization and bindings are preserved;
+the visible contract must authorize the checker identity, digest, or Jacobian
+record type. Otherwise claim the highest lower permitted assurance (`CHECKED` or
+`COMPUTED`), even if Jacobian returned `VERIFIED`; never reconstruct or
+paraphrase such a record from tool fields or transfer verification between
+claims or artifacts.
 For locally constructed inline input, check payload fields against the intended
 object. When output echoes scope or a bound digest/URI, compare it with the
 submitted input; do not use mismatched output. This catches routing and

benchmarks/datasets/mathematical-benchmarks-v1/jobs/jacobian-observation-proxy.json

@@ -37,6 +37,9 @@
   "agents": [
     {
       "name": "codex",
+      "skills": [
+        ".agents/skills/jacobian-math"
+      ],
       "kwargs": {
         "web_search": "disabled"
       }

benchmarks/datasets/mathematical-benchmarks-v1/jobs/jacobian-observation.json

@@ -35,6 +35,9 @@
   "agents": [
     {
       "name": "codex",
+      "skills": [
+        ".agents/skills/jacobian-math"
+      ],
       "kwargs": {
         "web_search": "disabled"
       }

benchmarks/schemas/harbor-job.schema.json

@@ -79,6 +79,11 @@
         "properties": {
           "name": {"type": "string", "minLength": 1},
           "model_name": {"type": "string", "minLength": 1},
+          "skills": {
+            "type": "array",
+            "uniqueItems": true,
+            "items": {"type": "string", "minLength": 1}
+          },
           "kwargs": {
             "type": "object",
             "additionalProperties": false,

benchmarks/tooling/benchmark_contracts.py

@@ -213,6 +213,23 @@ def _validate_task(suite: Suite, task_dir: Path) -> list[str]:
     return failures
 
 
+def _remove_jacobian_skill(agents: Any) -> None:
+    if not isinstance(agents, list):
+        return
+    for agent in agents:
+        if not isinstance(agent, dict) or not isinstance(agent.get("skills"), list):
+            continue
+        remaining = [
+            skill
+            for skill in agent["skills"]
+            if skill != ".agents/skills/jacobian-math"
+        ]
+        if remaining:
+            agent["skills"] = remaining
+        else:
+            agent.pop("skills", None)
+
+
 def _observation_pair_failures() -> list[str]:
     treatment_path = (
         BENCHMARKS
@@ -246,12 +263,14 @@ def normalized(value: dict[str, Any]) -> dict[str, Any]:
                     for item in compose
                     if "jacobian-observation.compose.yaml" not in item
                 ]
+        _remove_jacobian_skill(copy.get("agents"))
         return copy
 
     if normalized(treatment) != normalized(control):
         return [
             "agent workflow control/treatment jobs differ outside the allowed "
-            "jobs_dir, Jacobian sidecar composition, and sidecar telemetry artifact"
+            "jobs_dir, Jacobian skill, sidecar composition, and sidecar telemetry "
+            "artifact"
         ]
     return []
 

benchmarks/tooling/codex_tool_context.py

@@ -64,14 +64,15 @@ def _visible_results(step: dict[str, Any]) -> dict[str, int]:
     return visible_by_call
 
 
-def _analyze_step(step: object) -> tuple[int, int, int, int, int, int]:
+def _analyze_step(step: object) -> tuple[int, int, int, int, int, int, int]:
     if not isinstance(step, dict):
-        return (0, 0, 0, 0, 0, 0)
+        return (0, 0, 0, 0, 0, 0, 0)
     visible_by_call = _visible_results(step)
     tool_calls = step.get("tool_calls", [])
     if not isinstance(tool_calls, list):
-        return (0, 0, 0, 0, 0, 0)
+        return (0, 0, 0, 0, 0, 0, 0)
     scan_count = scan_bytes = unbound_scan_count = tool_output_bytes = 0
+    jacobian_output_bytes = 0
     direct_find_references = direct_run_references = 0
     for call in tool_calls:
         if not isinstance(call, dict):
@@ -83,8 +84,12 @@ def _analyze_step(step: object) -> tuple[int, int, int, int, int, int]:
         source = arguments.get("input") if isinstance(arguments, dict) else None
         if call.get("function_name") != "exec" or not isinstance(source, str):
             continue
-        direct_find_references += source.count(_JACOBIAN_FIND)
-        direct_run_references += source.count(_JACOBIAN_RUN)
+        find_references = source.count(_JACOBIAN_FIND)
+        run_references = source.count(_JACOBIAN_RUN)
+        direct_find_references += find_references
+        direct_run_references += run_references
+        if find_references or run_references:
+            jacobian_output_bytes += visible
         if "ALL_TOOLS" in source:
             scan_count += 1
             scan_bytes += visible
@@ -95,6 +100,7 @@ def _analyze_step(step: object) -> tuple[int, int, int, int, int, int]:
         scan_bytes,
         unbound_scan_count,
         tool_output_bytes,
+        jacobian_output_bytes,
         direct_find_references,
         direct_run_references,
     )
@@ -108,6 +114,7 @@ def analyze_trajectory(path: Path) -> dict[str, Any]:
     scan_bytes = 0
     unbound_scan_count = 0
     tool_output_bytes = 0
+    jacobian_output_bytes = 0
     direct_find_references = 0
     direct_run_references = 0
 
@@ -117,8 +124,9 @@ def analyze_trajectory(path: Path) -> dict[str, Any]:
         scan_bytes += step_counts[1]
         unbound_scan_count += step_counts[2]
         tool_output_bytes += step_counts[3]
-        direct_find_references += step_counts[4]
-        direct_run_references += step_counts[5]
+        jacobian_output_bytes += step_counts[4]
+        direct_find_references += step_counts[5]
+        direct_run_references += step_counts[6]
 
     metrics = trajectory.get("final_metrics")
     metrics = metrics if isinstance(metrics, dict) else {}
@@ -137,6 +145,7 @@ def analyze_trajectory(path: Path) -> dict[str, Any]:
         "all_tools_model_visible_bytes": scan_bytes,
         "all_tools_unbound_observation_count": unbound_scan_count,
         "tool_model_visible_bytes": tool_output_bytes,
+        "direct_jacobian_find_run_model_visible_bytes": jacobian_output_bytes,
         "direct_jacobian_find_references": direct_find_references,
         "direct_jacobian_run_references": direct_run_references,
         "prompt_tokens": prompt_tokens,
@@ -165,6 +174,33 @@ def build_report(paths: list[Path], *, label: str) -> dict[str, Any]:
         "label": label,
         "trial_count": len(trials),
         "summary": {
+            "jacobian_invocation_trials": sum(
+                int(
+                    trial["direct_jacobian_find_references"] > 0
+                    or trial["direct_jacobian_run_references"] > 0
+                )
+                for trial in trials
+            ),
+            "jacobian_execution_trials": sum(
+                int(trial["direct_jacobian_run_references"] > 0) for trial in trials
+            ),
+            "jacobian_unused_trials": sum(
+                int(
+                    trial["direct_jacobian_find_references"] == 0
+                    and trial["direct_jacobian_run_references"] == 0
+                )
+                for trial in trials
+            ),
+            "direct_jacobian_find_references": sum(
+                int(trial["direct_jacobian_find_references"]) for trial in trials
+            ),
+            "direct_jacobian_run_references": sum(
+                int(trial["direct_jacobian_run_references"]) for trial in trials
+            ),
+            "direct_jacobian_find_run_model_visible_bytes": sum(
+                int(trial["direct_jacobian_find_run_model_visible_bytes"])
+                for trial in trials
+            ),
             "all_tools_scan_trials": sum(
                 int(trial["all_tools_scan_count"] > 0) for trial in trials
             ),

benchmarks/tooling/observation_results.py

@@ -192,6 +192,15 @@ def _comparison_job(job: dict[str, Any]) -> dict[str, Any]:
             ]
     for agent in normalized.get("agents", []):
         if isinstance(agent, dict):
+            skills = agent.get("skills")
+            if isinstance(skills, list):
+                remaining_skills = [
+                    skill for skill in skills if skill != ".agents/skills/jacobian-math"
+                ]
+                if remaining_skills:
+                    agent["skills"] = remaining_skills
+                else:
+                    agent.pop("skills", None)
             servers = agent.get("mcp_servers")
             if isinstance(servers, list):
                 remaining = [

benchmarks/validation/conftest.py

@@ -5,6 +5,21 @@
 import sys
 from pathlib import Path
 
+import pytest
+
 ROOT = Path(__file__).parents[2]
 if str(ROOT) not in sys.path:
     sys.path.insert(0, str(ROOT))
+
+
+@pytest.fixture(autouse=True)
+def _isolate_task_verifier_imports():
+    """Prevent one task's top-level verifier_support import leaking to another."""
+
+    original_path = list(sys.path)
+    sys.modules.pop("verifier_support", None)
+    try:
+        yield
+    finally:
+        sys.modules.pop("verifier_support", None)
+        sys.path[:] = original_path

benchmarks/validation/test_observation_comparison.py

@@ -199,6 +199,7 @@ def test_comparison_normalization_allows_only_frozen_jacobian_differences() -> N
         "agents": [
             {
                 "name": "codex",
+                "skills": [".agents/skills/jacobian-math"],
                 "mcp_servers": [
                     {
                         "name": "jacobian",
@@ -216,6 +217,10 @@ def test_comparison_normalization_allows_only_frozen_jacobian_differences() -> N
     assert _comparison_job(control) != _comparison_job(treatment)
     treatment["artifacts"].pop()
 
+    treatment["agents"][0]["skills"].append("unexpected-skill")
+    assert _comparison_job(control) != _comparison_job(treatment)
+    treatment["agents"][0]["skills"].pop()
+
     heldout_treatment = {
         "artifacts": ["/logs/agent/trajectory.json"],
         "environment": {
@@ -227,6 +232,7 @@ def test_comparison_normalization_allows_only_frozen_jacobian_differences() -> N
         "agents": [
             {
                 "name": "codex",
+                "skills": [".agents/skills/jacobian-math"],
                 "mcp_servers": [
                     {
                         "name": "jacobian",