fix(benchmarks): harden Jacobian evaluation evidence (#946)

Author: N0zoM1z0

.agents/skills/jacobian-math/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: jacobian-math
-description: Use Jacobian for exact mathematics, matrix determinants, symbolic computation, structural analysis, counterexamples, bounded search, and independent verification. Trigger on relevant math tasks even when the user does not name Jacobian.
+description: Use Jacobian for exact math, matrix determinants, symbolic work, counterexamples, search, and independent verification, even when unnamed.
 ---
 
 # Jacobian Math
@@ -10,19 +10,20 @@ description: Use Jacobian for exact mathematics, matrix determinants, symbolic c
 In Code Mode, call
 `tools.mcp__jacobian__math_find(...)` and
 `tools.mcp__jacobian__math_run(...)` directly. Do not enumerate, filter, or
-print `ALL_TOOLS`. Return the typed projection when available:
+print `ALL_TOOLS`. Return the typed projection:
 
 ```js
 const r = await tools.mcp__jacobian__math_find({query: "...", limit: 3});
 text(r.structuredContent ?? r);
 ```
 
-Use Jacobian for each requested exact mathematical outcome, even when small.
+Use Jacobian for each requested exact outcome, even when small.
 Keep decomposition and routing decisions agent-owned; composing already-known
 supporting operations remains allowed when clearer.
 Do not call Jacobian for definitions, formatting, or non-execution tasks.
 
-Call `math.run` directly for these stable contracts, preserving JSON types:
+No discovery for stable producers: `{"capability_id":"<id>","mode":"EXPLORE","payload":<JSON>}`
+(not `COMPUTE`/`input`). Payloads:
 
 - `integer.compute.gcd`, `integer.compute.lcm`, or `integer.compute.extended_gcd`:
   `{"left":"84","right":"30"}`.
@@ -35,38 +36,38 @@ Call `math.run` directly for these stable contracts, preserving JSON types:
   `polynomial.expression.normalize` contract directly.
 - `matrix.determinant.verify` in `VERIFY` mode for an independent check:
   `{"determinant_uri":"<determinant_uri from compute output>"}`.
+- `combinatorics.cyclic_difference_set.extension.decide`:
+  `{"base_elements":["1","2","4","8","13"],"target_order":7}`.
+  `combinatorics.cyclic_difference_set.extension.verify` uses
+  `{"input":<same payload>,"candidate":<producer output.result>}` in `VERIFY` mode.
 
-For other outcomes, use `math.find` with a specific plain-language outcome;
-no capability ID is required.
-Use low `limit` values. For a selected operation's schema, call
+For other outcomes, query `math.find` by plain-language outcome; no ID is
+required. Use low `limit` only for query search, never with `capability_id`. For
+a selected operation's schema, call
 `math.find({"capability_id":"<exact-id>","view":"CONTRACT"})`; never send
 `mode: "CONTRACT"` to `math.run` or put `CONTRACT` in a query. A card's
 `invocation_example`, or required top-level fields, may be enough.
 
-Do not add a discovery domain filter unless its exact installed spelling is
-known. Follow exposed recovery paths such as removing unknown filters or
-reformulating the query before
-treating absence as final. After invalid input, correct the reported constraint
-and retry within the task resource bounds. If
-one provider is unavailable, continue with other installed routes that can
-produce the outcome. Treat timeouts, cancellations, errors, incomplete searches,
-and missing witnesses as non-conclusions. Accept only a completed
-result whose scope covers the input, and carry forward the smallest decisive
-value, witness, status, assurance, completeness, and open obligations; preserve
-artifact refs, including verification record URIs.
+Add no domain filter unless its installed spelling is known. Follow exposed
+recovery paths by removing unknown filters or reformulating the query. After
+invalid input, correct the constraint and retry within the task resource bounds.
+If one provider is unavailable, continue with other installed routes. Treat
+timeouts, cancellations, errors, incomplete searches, and missing witnesses as
+non-conclusions. Accept only completed results covering the input; carry forward
+the smallest decisive value, witness, status, assurance, completeness, and open
+obligations plus artifact and verification-record URIs.
 
 Keep representation, decomposition, composition, iteration, verification
 timing, and stopping decisions agent-owned.
 
-Model-authored calculations or programs are not independent evidence. Use
-installed `VERIFY` when requested; a writable path or schema alone is not
-authorization. For task-level `VERIFIED`, require result assurance `VERIFIED`,
-exact record bytes, and that required task authorization and bindings are preserved;
-the visible contract must authorize the checker identity, digest, or Jacobian
-record type. Otherwise claim the highest lower permitted assurance (`CHECKED` or
-`COMPUTED`), even if Jacobian returned `VERIFIED`; never reconstruct or
-paraphrase such a record from tool fields or transfer verification between
-claims or artifacts.
+Model-authored work is not independent evidence. Use installed `VERIFY` when
+requested; a writable path or schema alone is not authorization. Task-level
+`VERIFIED` requires exact record bytes, result assurance `VERIFIED`, required
+task authorization and bindings are preserved, and a contract-authorized
+checker identity, digest, or Jacobian record type. Otherwise claim the highest
+lower permitted assurance (`CHECKED` or `COMPUTED`), even if Jacobian returned
+`VERIFIED`; never
+reconstruct or paraphrase such a record or transfer it between claims.
 For locally constructed inline input, check payload fields against the intended
 object. When output echoes scope or a bound digest/URI, compare it with the
 submitted input; do not use mismatched output. This catches routing and

.github/workflows/benchmarks.yml

@@ -310,7 +310,7 @@ jobs:
           name: benchmark-oracle-${{ matrix.dataset }}-${{ matrix.shard }}
           path: |
             benchmarks/results/${{ matrix.dataset }}-oracle
-            benchmarks/results/${{ matrix.dataset }}-oracle/jacobian-augmented-task-digests.json
+            benchmarks/results/${{ matrix.dataset }}-oracle/jacobian-augmented-task-digests.*.json
             benchmarks/results/${{ matrix.dataset }}-oracle/**/oracle-evidence.json
           if-no-files-found: ignore
           retention-days: ${{ github.event_name == 'pull_request' && 14 || 90 }}

benchmarks/config/codex-visibility-v2.json

@@ -56,6 +56,15 @@
       ],
       "require_verified": false
     },
+    {
+      "case_id": "latent-fixed-order-difference-set-extension",
+      "cue_level": "LATENT",
+      "prompt": "For the integer set {1,2,4,8,13}, decide exactly whether its reduced residues are contained in a cyclic perfect difference set of order 7, and report the exact candidate-space size covered by the decision.",
+      "expected_capability_ids": [
+        "combinatorics.cyclic_difference_set.extension.decide"
+      ],
+      "require_verified": false
+    },
     {
       "case_id": "latent-independent-verification",
       "cue_level": "LATENT",

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-013/environment/submission_schema.json

@@ -47,14 +47,19 @@
           "minLength": 1
         },
         "key_facts": {
-          "additionalProperties": {
-            "type": "string",
-            "minLength": 1
-          },
-          "minProperties": 1,
-          "propertyNames": {
-            "pattern": "^[a-z][a-z0-9_]*$"
+          "additionalProperties": false,
+          "properties": {
+            "identity_holds": {
+              "const": "true"
+            },
+            "index_order": {
+              "const": "a<a+1<2a<2a+2<c<c+1"
+            }
           },
+          "required": [
+            "identity_holds",
+            "index_order"
+          ],
           "type": "object"
         }
       }

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-013/instruction.md

@@ -8,9 +8,9 @@ report the capability boundary honestly. The portfolio does not currently own a
 
 Write `submission.json` to the exact agent-visible `submission_schema.json`, record your
 reasoning in `evidence/answer.txt`, and include that file's SHA-256 digest in the evidence
-list. In `result.key_facts`, use a nonempty object with lower-snake-case fact
-names and nonempty string values; do not rely on prose or numeric/boolean
-coercion. You may claim `UNVERIFIED` or `COMPUTED`; only `COMPUTED`
+list. In `result.key_facts`, reproduce the exact public facts declared by
+`submission_schema.json`; do not rename the fields or coerce their string
+values to booleans. You may claim `UNVERIFIED` or `COMPUTED`; only `COMPUTED`
 receives full aggregate credit. Do not claim `VERIFIED`. Treat timeout,
 error, or incomplete search as a non-conclusion, not as evidence for or
 against the claim.

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-013/task.toml

@@ -2,7 +2,7 @@ schema_version = "1.4"
 artifacts = ["/app/submission.json", "/app/evidence"]
 [task]
 name = "jacobian/jcb-postdoc-013"
-version = "1.0.0"
+version = "1.1.0"
 description = "Research diagnostic: Erdős Problem 397: a parametric family of central-binomial collisions"
 keywords = ["research", "diagnostic", "mathematical-sciences", "enumerative combinatorics", "number theory", "symbolic identities"]
 [metadata]

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-013/tests/public_contract.json

@@ -65,14 +65,19 @@
             ]
           },
           "key_facts": {
-            "additionalProperties": {
-              "minLength": 1,
-              "type": "string"
-            },
-            "minProperties": 1,
-            "propertyNames": {
-              "pattern": "^[a-z][a-z0-9_]*$"
+            "additionalProperties": false,
+            "properties": {
+              "identity_holds": {
+                "const": "true"
+              },
+              "index_order": {
+                "const": "a<a+1<2a<2a+2<c<c+1"
+              }
             },
+            "required": [
+              "identity_holds",
+              "index_order"
+            ],
             "type": "object"
           }
         },

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-015/instruction.md

@@ -15,9 +15,10 @@ checkable finite core:
 - the exact number of candidate supersets examined for each fixed order.
 
 Write that evidence to `evidence/finite-core.json` and bind its SHA-256 digest
-in `submission.json`. Follow the exact schemas in `input.json` and
-`submission_schema.json`. The fixed-order checks must cover every candidate,
-not a sample. The public universal result may be reported, but this task does
+in `submission.json`. Follow `submission_schema.json` for the submission and
+the separate agent-visible `/app/evidence_schema.json` for the complete
+`evidence/finite-core.json` body. The fixed-order checks must cover every
+candidate, not a sample. The public universal result may be reported, but this task does
 not supply its projective-geometric or formal obstruction certificate: set
 `universal_obstruction_replayed` to `false` and do not claim that the three
 finite searches prove the universal statement.

benchmarks/datasets/research-diagnostics-v1/jcb-postdoc-015/task.toml

@@ -2,7 +2,7 @@ schema_version = "1.4"
 artifacts = ["/app/submission.json", "/app/evidence"]
 [task]
 name = "jacobian/jcb-postdoc-015"
-version = "2.0.0"
+version = "2.1.0"
 description = "Research diagnostic: Erdős Problem 707 finite Sidon and fixed-order extension evidence"
 keywords = ["research", "diagnostic", "mathematical-sciences", "additive combinatorics", "finite geometry", "modular arithmetic"]
 [metadata]