fix(benchmarks): remove hidden Farkas evidence cap - #1333
Draft
kaoru0822-kitauji wants to merge 1 commit into
Draft
fix(benchmarks): remove hidden Farkas evidence cap#1333kaoru0822-kitauji wants to merge 1 commit into
kaoru0822-kitauji wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is one bounded benchmark-integrity change based on an audit of historical PR #876 against the current Harbor architecture at
413e3846c2529af9e8f90a96d6f5679b23ed04f2.The audited Farkas task accepted schema-valid evidence with arbitrary JSON whitespace in its public contract, but the hidden verifier rejected files above 1 MiB. This removes that unpublished cap and ports the task-local loader to the repository's existing streaming JSON pattern. No product/runtime behavior, evaluator strategy, or general agent skill is added.
Integrity audit
exact-farkas-ldl-slicehad a hidden 1 MiB cap. This PR fixes that one task.task.toml; timeout/error states remain non-conclusions.Frozen task sample
Source SHA:
413e3846c2529af9e8f90a96d6f5679b23ed04f2exact-farkas-ldl-slice:sha256:5ff1eed2a5ff443222d181f3c676ac5012b4cd01189f1d983ef2e5256e447848polynomial-map-collision:sha256:9243ca8ee4800cb8eae039a08e0414c7fcfd8e111c3d1a3c8d6fbbfc44884abdrp2-homology-lattice:sha256:35c60550453aa1979b4694a399813023844ab3a6524643e106a9e983836b6539apollonius-gap-repair:sha256:701b5fc809ec6973a55d8a30568d89829c9c7f683b80c04241a4df2d6bc61636elementwise-fixed-no-global-invariant:sha256:ce3612dd2ebff71eb14234c965ef1d8959e19e43f0953b3366df13f04446f2aeProspective changed-task digest:
sha256:174ddb8eb0f35a81f21a196aaa7dc562f5af399278da763b5bc6f7f5c0bf2e58.Pilot evidence
The planned paired pilot was gpt-5.6-luna at medium reasoning, with a no-Jacobian control and a current-Jacobian-MCP treatment, and no general evaluation skill exposed. Both substantive attempts ran in named tmux sessions.
The host has no Docker daemon, so a Bubblewrap clean room was used after proving that the repository and verifier files were not visible. The attempt was stopped after repeated infrastructure failures: the installed standalone Codex package lacked its code-mode host executable inside the clean room, so the model could not read or write the task workspace. The control produced only a blocker trajectory (59,307 input, 52,992 cached input, 552 output, 142 reasoning tokens); neither condition produced a mathematical submission. The comparison is therefore INCONCLUSIVE, not a treatment result. It provides no evidence for adding a general tool-use skill, and this PR adds none.
Change
MAX_EVIDENCE_BYTESrejection.Validation
make harbor-plan BASE=origin/main: selects only the Farkas host leaf, its generic verifier-contract leaf, and its Oracle.git diff --check origin/main...HEAD: passed.make check: Ruff, formatting, complexity, mypy, and 869/870 unit tests passed. The remaining unrelatedtest_python_distribution_identity_binds_installed_file_bytesfailed due same-timestamp metadata caching; its isolated reproduction failed once and passed immediately on retry without a code change, and the full retry hit the same pre-existing timing condition./bin/sh: docker: not found), so no Oracle verdict is claimed.Deferred proof gaps
A Docker-capable host should run the selected exact-task Oracle and the frozen five-task paired pilot. Those are explicit evidence gaps; they do not broaden this PR beyond the single reproducible verifier-contract defect.