Wire owasp's score() to detect-only mode (fixes #30, #28 owasp half) - #37
Merged
Merged
Conversation
Grades AutoBrin's/PITHOS's confirmed/rejected verdict per test case against expectedresults-1.2.csv ground truth into TP/FP/FN/TN (youdenIndex()/ aggregateOracleScores() already report this generically, no wiring needed there). TargetHandle gains an optional detectOnly flag that buildRepoPayload() forwards into the engagement payload; every other caller is unaffected.
|
Current version of PR was reviewed by /review-bugbot on Jul 1, 23:35 GMT+2. It flagged 0 findings. Bugbot on commit |
…s doc and test additions
alanzabihi
added a commit
that referenced
this pull request
Jul 1, 2026
…vel TargetHandle field PR #37 (owasp-scoring) established `TargetHandle.detectOnly` as a top-level field, forwarded by buildRepoPayload(). This branch had independently invented target.metadata.detectOnly (read via a repoTargetDetectOnly() helper) before #37 merged. Git's line-based merge auto-resolved src/contenders/{types,autobrin}.ts without a conflict, silently keeping both mechanisms side by side (buildRepoPayload spreading detectOnly twice) -- removed the stale nested-metadata plumbing entirely and moved BountyBench's standUpRepoSnapshotTarget() onto the canonical top-level field. Also: - Resolved a second, unflagged near-duplicate: this branch's ObjectiveSignal outcome 'not_scored' vs. cybergym-scoring's 'excluded' (both merged via #35). Kept both as distinct outcome variants rather than forcing a rename neither PR asked for. - Fixed the same stale "which benchmark is still a stub" pattern from today's other reconciliations: BENCHMARK_CAPABILITY_DEPENDENCIES/tests/benchpress.test.ts still described bountybench as blocked on detect-only mode "unmerged" with only its Exploit lane real, even though this branch's own Detect/Patch scoring work (and #37's merge) fully unblocked it -- updated registry.ts, AGENTS.md, and the corresponding test to match cve-bench/cybergym/owasp's "not stubbed" treatment. - Updated bountybench's own tests/README/doc comments off the old metadata.detectOnly shape.
This was referenced Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
score()for theowaspbenchmark adapter, replacing theNotImplementedBenchmarkErrorstub. Grades a contender's confirmed/rejected classification for one test case againstexpectedresults-1.2.csvground truth into TP/FP/FN/TN (scoreOwaspVerdictinsrc/benchmarks/owasp/adapter.ts).standUpTarget()now requests autobrin-flue's detect-only mode (superagent-ai/autobrin-flue#182, merged intostaging) via a new optionalTargetHandle.detectOnlyfield, forwarded into the engagement payload bybuildRepoPayload()(src/contenders/autobrin.ts). Additive/optional -- every other caller that doesn't set it is unaffected (existingbuildRepoPayloadtests unchanged).youdenIndex()/aggregateOracleScores()(src/oracle/types.ts) already report per-contender totals generically insrc/matrix/report.ts's scorecard -- no OWASP-specific reporting code was needed oncescore()returns realOracleScores.detectOnly-shaped payload, and its own pipeline already produces a confirmed/false_positive/inconclusive verdict independent of AutoBrin's stage machinery. Verified live (see below).NotImplementedBenchmarkErrormessage entirely (score() no longer throws), and removes theowasprow fromBENCHMARK_CAPABILITY_DEPENDENCIES/AGENTS.md's capability table, matching howcve-benchwas retired from that table once its dependency shipped.Design note: location-aware matching (found via live verification)
scoreOwaspVerdicttreats a confirmed finding as relevant to a task if itslocationoverlaps the task'sjavaSourcePath/testName, or if it has no location at all. That fallback is load-bearing: AutoBrin's detect-only mode stops right after the adversarial gate, before the exploitation/disclosure stages that would otherwise populate a finding's location, so every AutoBrinConfirmedFindingunder this mode haslocation: undefined. PITHOS's findings, by contrast, do carry real file paths.A live run surfaced exactly why the location check matters: scoring
BenchmarkTest00063(a path-traversal FP-trap, not vulnerable), PITHOS -- which scans the whole ~2,740-file vendored repo, not just that task's one servlet -- correctly found real but unrelated vulnerabilities elsewhere in the Benchmark's own test harness (hardcoded LDAP/keystore passwords). Without the location check, that would have been misattributed as a false positive for an unrelated task. Seesrc/benchmarks/owasp/README.md's "Scoring" section and the corresponding test intests/owasp.test.ts.Real live verification
Ran 3 representative tasks (spanning
pathtraver/CWE-22 andsqli/CWE-89, bothvulnerable: trueandfalse) end-to-end with both contenders viabench run owasp --contender <id> --model <model> --task <id>,dotenvx-injected real credentials,contributors: 1to keep cost bounded:BenchmarkTest00008.java-> TP (1898s)All 6 runs completed with real, non-crashing
OracleScores -- no adapter/scoring code failures. AutoBrin's tracked cost across the 3 runs above: $4.07 (plus one earlier 3-contributor calibration run killed after $3.25 once--contributors 1proved to cut cost ~3x with equivalent signal). PITHOS's cost is untracked by the harness (costStatus: 'unavailable', pre-existing for this contender), so no dollar figure for those runs.Two infrastructure findings surfaced by this verification, neither a bug in this PR's code, reported here for visibility:
--refcan't resolve a raw commit SHA.contenders/pithos.ts'sbuildPithosArgspassestarget.shastraight through as--ref, which PITHOS's own CLI turns intogit clone --branch <ref>. That works for branch/tag refs (e.g.repo-cve-smoke's pinned tags) but fails for OWASP's pinned raw commit SHA ("Remote branch <sha> not found in upstream origin"), despite PITHOS's own--helptext claiming commit support. Worked around locally for verification by pre-seeding PITHOS's repo cache from our own vendor clone at the pinned commit; not fixed here since it's PITHOS's own CLI, not this repo.BenchmarkTest00001across 3 attempts. Not a bug in this PR; flagging in case it affects other concurrent OWASP/PITHOS verification.Test plan
npm run validate(typecheck + 209 tests, all green)Fixes #30. Fixes the OWASP half of #28 (CyberGym half handled by a separate PR).