Wire bountybench Detect/Patch lane scoring; verify PITHOS on the Exploit lane

[alanzabihi]

Status today

Two separate gaps in this one benchmark:

1. Detect/Patch lanes: score() blocked for everyone

standUpTarget() works for detect/patch tasks (modality: 'repo', real codebase clone), but score() (src/benchmarks/bountybench/adapter.ts:276, guarded by BountyBenchScoreBlockedError at line 19) unconditionally throws:

BountyBench {detect,patch} scoring for task "..." is blocked on autobrin-flue#182 (detect-only mode + proposed_patch disclosure output) -- not implemented.

Both dependencies are now merged into staging via autobrin-flue#182 (PR #186): detectOnly payload flag, and proposed_patch disclosure output ({ summary, diff, files }, host-validated with git apply --check).

What's needed:

• Detect lane (autobrin): same shape as #30's OWASP wiring -- set detectOnly: true, read the confirmed/rejected verdict, compare against BountyBench's own known-vulnerable ground truth per bounty.
• Patch lane (autobrin): run against the vulnerable codebase, take the disclosure's proposed_patch.diff, apply it to a fresh checkout, and re-verify the vulnerability is actually gone (a patched-artifact check in the same spirit as the differential oracle, but validating the contender's own patch rather than a public fix commit).
• PITHOS on Detect lane: likely works the same way as OWASP (Wire owasp's score() to the merged detect-only mode #30) -- PITHOS's own verdict needs no detectOnly-equivalent flag.
• PITHOS on Patch lane: needs a design decision. PITHOS's TRIAGE.json findings have no patch/diff field today -- confirm whether PITHOS has (or should grow) a patch-proposal capability, or whether the Patch lane stays autobrin-only.

2. Exploit lane: score() already works, but only tested with autobrin

Unlike CVE-Bench, BountyBench's TargetHandle (both lanes) sets a top-level target.repo (metadata.codebaseRepo) alongside the nested metadata.webapp shape (buildExploitTargetHandle(), src/benchmarks/bountybench/adapter.ts:125). PITHOS's contender only checks target.repo exists, not modality -- so it should be able to run against this lane today, unlike CVE-Bench where no repo field exists at all.

This has never actually been run: PITHOS would clone and statically analyze the codebase, never touching the live webapp standUpTarget() stands up, while autobrin dynamically exploits it -- same target, different methodology. Needs a real live run to confirm:

• PITHOS's contender actually completes against a bounty task's target.repo.
• buildExploitOracleScore()'s live per-vuln verifier check works generically against PITHOS's ContenderClaim shape, not just autobrin's.
• The already-running Docker stack (stood up for the shared task, before PITHOS's static analysis begins) doesn't need to be up at all for PITHOS's run -- wasted setup cost worth knowing about when budgeting a real comparison run.

[alanzabihi]

Update on the detectOnly payload-threading pattern that #33's PR description flagged for reconciliation:

#37 (owasp-scoring, since merged to main) established the canonical shape: a top-level TargetHandle.detectOnly field, alongside modality/repo/sha — not nested in the generic metadata bag. #33 had independently invented target.metadata.detectOnly before #37 merged (there was no open OWASP-scoring PR to reuse a convention from at the time).

#33 is now rebased onto latest main and reconciled onto #37's top-level field. Full details — plus two related things found and fixed along the way (a second, unflagged not_scored/excluded outcome near-duplicate with #35's cybergym-scoring work, a stale "bountybench is still partially stubbed" test, and a real Bugbot-caught patch-scoring determinism/correctness bug) — are in this comment on #33.

Fresh live re-verification: the reconciled autobrin@staging Detect-lane engagement against parse-url-0-detect still forwards detectOnly: true correctly end to end (top-level field confirmed live, not nested in metadata) — 353s, $0.627, well under budget. npm run validate is fully green (258 tests). Not merged yet; #33 is up to date and ready for review.