Skip to content

fix(benchmarks): remove hidden Farkas evidence cap - #1333

Draft
kaoru0822-kitauji wants to merge 1 commit into
morluto:mainfrom
kaoru0822-kitauji:agent/harbor-integrity-pilot
Draft

fix(benchmarks): remove hidden Farkas evidence cap#1333
kaoru0822-kitauji wants to merge 1 commit into
morluto:mainfrom
kaoru0822-kitauji:agent/harbor-integrity-pilot

Conversation

@kaoru0822-kitauji

@kaoru0822-kitauji kaoru0822-kitauji commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

This is one bounded benchmark-integrity change based on an audit of historical PR #876 against the current Harbor architecture at 413e3846c2529af9e8f90a96d6f5679b23ed04f2.

The audited Farkas task accepted schema-valid evidence with arbitrary JSON whitespace in its public contract, but the hidden verifier rejected files above 1 MiB. This removes that unpublished cap and ports the task-local loader to the repository's existing streaming JSON pattern. No product/runtime behavior, evaluator strategy, or general agent skill is added.

Integrity audit

Invariant Current assessment
Hidden task material is isolated Already solved: Harbor runs agents and verifiers in separate Docker environments, with static guards against hidden copies, widened build contexts, visibility leaks, and symlink escapes.
Revisions are reachable and immutable Already solved: source SHA, clean snapshot identity, task digests, and observation records bind current reachable Git trees.
Terminal labels do not depend on a reasoning-log protocol Historical/obsolete: terminal states come from the exact task verifier; telemetry is passive and has no verdict authority.
Evidence has no hidden size limit Reproducibly broken here: exact-farkas-ldl-slice had a hidden 1 MiB cap. This PR fixes that one task.
Equivalent encodings are accepted Solved for the audited tasks through explicit canonical rules or semantic comparison.
Limits are task-owned Solved: Harbor task limits live in task.toml; timeout/error states remain non-conclusions.
Every validity rule is public Partially solved before this PR because the Farkas size cap was unpublished; fixed for this task.
Alternate valid witnesses are accepted Solved in the sampled tasks (including alternate LDL/Sylvester evidence, RP2 witnesses, primes, and prose).
Malformed/false assurance fails closed Solved through task-local and generic adversarial matrices.
Local versus generic verifier ownership is clear Solved: task semantics stay local; generic metadata/integrity attacks remain in the shared matrix.

Frozen task sample

Source SHA: 413e3846c2529af9e8f90a96d6f5679b23ed04f2

  • exact-farkas-ldl-slice: sha256:5ff1eed2a5ff443222d181f3c676ac5012b4cd01189f1d983ef2e5256e447848
  • polynomial-map-collision: sha256:9243ca8ee4800cb8eae039a08e0414c7fcfd8e111c3d1a3c8d6fbbfc44884abd
  • rp2-homology-lattice: sha256:35c60550453aa1979b4694a399813023844ab3a6524643e106a9e983836b6539
  • apollonius-gap-repair: sha256:701b5fc809ec6973a55d8a30568d89829c9c7f683b80c04241a4df2d6bc61636
  • elementwise-fixed-no-global-invariant: sha256:ce3612dd2ebff71eb14234c965ef1d8959e19e43f0953b3366df13f04446f2ae

Prospective changed-task digest: sha256:174ddb8eb0f35a81f21a196aaa7dc562f5af399278da763b5bc6f7f5c0bf2e58.

Pilot evidence

The planned paired pilot was gpt-5.6-luna at medium reasoning, with a no-Jacobian control and a current-Jacobian-MCP treatment, and no general evaluation skill exposed. Both substantive attempts ran in named tmux sessions.

The host has no Docker daemon, so a Bubblewrap clean room was used after proving that the repository and verifier files were not visible. The attempt was stopped after repeated infrastructure failures: the installed standalone Codex package lacked its code-mode host executable inside the clean room, so the model could not read or write the task workspace. The control produced only a blocker trajectory (59,307 input, 52,992 cached input, 552 output, 142 reasoning tokens); neither condition produced a mathematical submission. The comparison is therefore INCONCLUSIVE, not a treatment result. It provides no evidence for adding a general tool-use skill, and this PR adds none.

Change

  • Remove the unpublished MAX_EVIDENCE_BYTES rejection.
  • Stream one JSON value while discarding arbitrarily large legal leading/trailing whitespace.
  • Continue to reject leading/trailing non-whitespace garbage and duplicate JSON values.
  • Add a regression above 1 MiB plus adversarial garbage cases.
  • Refresh the task verifier checksum.

Validation

  • make harbor-plan BASE=origin/main: selects only the Farkas host leaf, its generic verifier-contract leaf, and its Oracle.
  • Focused task leaf: 10 passed.
  • Generic verifier-contract leaf: 12 passed, 1,658 deselected.
  • Static task quality and contract checks: passed.
  • git diff --check origin/main...HEAD: passed.
  • make check: Ruff, formatting, complexity, mypy, and 869/870 unit tests passed. The remaining unrelated test_python_distribution_identity_binds_installed_file_bytes failed due same-timestamp metadata caching; its isolated reproduction failed once and passed immediately on retry without a code change, and the full retry hit the same pre-existing timing condition.
  • Exact Harbor Oracle: not run because Docker is absent (/bin/sh: docker: not found), so no Oracle verdict is claimed.

Deferred proof gaps

A Docker-capable host should run the selected exact-task Oracle and the frozen five-task paired pilot. Those are explicit evidence gaps; they do not broaden this PR beyond the single reproducible verifier-contract defect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant