Skip to content

Wire owasp's score() to detect-only mode (fixes #30, #28 owasp half) - #37

Merged
alanzabihi merged 2 commits into
mainfrom
owasp-scoring
Jul 1, 2026
Merged

alanzabihi merged 2 commits into
mainfrom
owasp-scoring

Conversation

@alanzabihi

Copy link
Copy Markdown
Contributor

Summary

  • Implements real score() for the owasp benchmark adapter, replacing the NotImplementedBenchmarkError stub. Grades a contender's confirmed/rejected classification for one test case against expectedresults-1.2.csv ground truth into TP/FP/FN/TN (scoreOwaspVerdict in src/benchmarks/owasp/adapter.ts).
  • standUpTarget() now requests autobrin-flue's detect-only mode (superagent-ai/autobrin-flue#182, merged into staging) via a new optional TargetHandle.detectOnly field, forwarded into the engagement payload by buildRepoPayload() (src/contenders/autobrin.ts). Additive/optional -- every other caller that doesn't set it is unaffected (existing buildRepoPayload tests unchanged).
  • youdenIndex()/aggregateOracleScores() (src/oracle/types.ts) already report per-contender totals generically in src/matrix/report.ts's scorecard -- no OWASP-specific reporting code was needed once score() returns real OracleScores.
  • PITHOS needed no adapter-side changes: it never takes a detectOnly-shaped payload, and its own pipeline already produces a confirmed/false_positive/inconclusive verdict independent of AutoBrin's stage machinery. Verified live (see below).
  • Fixes cybergym/owasp score() error messages say capabilities are unmerged, but they merged today #28's OWASP half by removing the stale NotImplementedBenchmarkError message entirely (score() no longer throws), and removes the owasp row from BENCHMARK_CAPABILITY_DEPENDENCIES / AGENTS.md's capability table, matching how cve-bench was retired from that table once its dependency shipped.

Design note: location-aware matching (found via live verification)

scoreOwaspVerdict treats a confirmed finding as relevant to a task if its location overlaps the task's javaSourcePath/testName, or if it has no location at all. That fallback is load-bearing: AutoBrin's detect-only mode stops right after the adversarial gate, before the exploitation/disclosure stages that would otherwise populate a finding's location, so every AutoBrin ConfirmedFinding under this mode has location: undefined. PITHOS's findings, by contrast, do carry real file paths.

A live run surfaced exactly why the location check matters: scoring BenchmarkTest00063 (a path-traversal FP-trap, not vulnerable), PITHOS -- which scans the whole ~2,740-file vendored repo, not just that task's one servlet -- correctly found real but unrelated vulnerabilities elsewhere in the Benchmark's own test harness (hardcoded LDAP/keystore passwords). Without the location check, that would have been misattributed as a false positive for an unrelated task. See src/benchmarks/owasp/README.md's "Scoring" section and the corresponding test in tests/owasp.test.ts.

Real live verification

Ran 3 representative tasks (spanning pathtraver/CWE-22 and sqli/CWE-89, both vulnerable: true and false) end-to-end with both contenders via bench run owasp --contender <id> --model <model> --task <id>, dotenvx-injected real credentials, contributors: 1 to keep cost bounded:

Task Ground truth autobrin@staging (kimi-azure/kimi-k2.6, detectOnly) pithos (kimi-k2.6, azure-openai-responses)
BenchmarkTest00001 (pathtraver) vulnerable rejected -> FN ($1.55, 811s) rate-limited to an empty claim -> FN (non-crashing, but not a meaningful signal; see below)
BenchmarkTest00063 (pathtraver) not vulnerable rejected -> TN ($1.65, 778s) confirmed 6 real-but-unrelated findings -> TN after the location fix above (555s)
BenchmarkTest00008 (sqli) vulnerable rejected -> FN ($0.87, 436s) confirmed a real SQL-injection finding that names BenchmarkTest00008.java -> TP (1898s)

All 6 runs completed with real, non-crashing OracleScores -- no adapter/scoring code failures. AutoBrin's tracked cost across the 3 runs above: $4.07 (plus one earlier 3-contributor calibration run killed after $3.25 once --contributors 1 proved to cut cost ~3x with equivalent signal). PITHOS's cost is untracked by the harness (costStatus: 'unavailable', pre-existing for this contender), so no dollar figure for those runs.

Two infrastructure findings surfaced by this verification, neither a bug in this PR's code, reported here for visibility:

  • PITHOS's --ref can't resolve a raw commit SHA. contenders/pithos.ts's buildPithosArgs passes target.sha straight through as --ref, which PITHOS's own CLI turns into git clone --branch <ref>. That works for branch/tag refs (e.g. repo-cve-smoke's pinned tags) but fails for OWASP's pinned raw commit SHA ("Remote branch <sha> not found in upstream origin"), despite PITHOS's own --help text claiming commit support. Worked around locally for verification by pre-seeding PITHOS's repo cache from our own vendor clone at the pinned commit; not fixed here since it's PITHOS's own CLI, not this repo.
  • Shared Azure Kimi-K2.6/eastus2 capacity is contended. Multiple parallel subagents were running concurrent kimi-azure engagements against the same deployment while this PR's live verification ran. AutoBrin's own provider-503/retry handling absorbed this fine; PITHOS's retry backoff (3 attempts, 2s/4s/8s) gave up faster under sustained 429s and fell back to empty "repair" results for BenchmarkTest00001 across 3 attempts. Not a bug in this PR; flagging in case it affects other concurrent OWASP/PITHOS verification.

Test plan

  • npm run validate (typecheck + 209 tests, all green)
  • Local Bugbot review of the uncommitted diff: no bugs found
  • Real live verification above (3 tasks x 2 contenders, all non-crashing)

Fixes #30. Fixes the OWASP half of #28 (CyberGym half handled by a separate PR).

Grades AutoBrin's/PITHOS's confirmed/rejected verdict per test case against
expectedresults-1.2.csv ground truth into TP/FP/FN/TN (youdenIndex()/
aggregateOracleScores() already report this generically, no wiring needed
there). TargetHandle gains an optional detectOnly flag that buildRepoPayload()
forwards into the engagement payload; every other caller is unaffected.
@cursor

cursor Bot commented Jul 1, 2026

Copy link
Copy Markdown

Current version of PR was reviewed by /review-bugbot on Jul 1, 23:35 GMT+2. It flagged 0 findings.

Bugbot on commit 59e7c39 is skipped.

@alanzabihi
alanzabihi merged commit d4116ec into main Jul 1, 2026
1 check passed
@alanzabihi
alanzabihi deleted the owasp-scoring branch July 1, 2026 21:56
alanzabihi added a commit that referenced this pull request Jul 1, 2026
…vel TargetHandle field

PR #37 (owasp-scoring) established `TargetHandle.detectOnly` as a top-level field, forwarded by
buildRepoPayload(). This branch had independently invented target.metadata.detectOnly (read via a
repoTargetDetectOnly() helper) before #37 merged. Git's line-based merge auto-resolved
src/contenders/{types,autobrin}.ts without a conflict, silently keeping both mechanisms side by
side (buildRepoPayload spreading detectOnly twice) -- removed the stale nested-metadata plumbing
entirely and moved BountyBench's standUpRepoSnapshotTarget() onto the canonical top-level field.

Also:
- Resolved a second, unflagged near-duplicate: this branch's ObjectiveSignal outcome 'not_scored'
  vs. cybergym-scoring's 'excluded' (both merged via #35). Kept both as distinct outcome variants
  rather than forcing a rename neither PR asked for.
- Fixed the same stale "which benchmark is still a stub" pattern from today's other reconciliations:
  BENCHMARK_CAPABILITY_DEPENDENCIES/tests/benchpress.test.ts still described bountybench as blocked
  on detect-only mode "unmerged" with only its Exploit lane real, even though this branch's own
  Detect/Patch scoring work (and #37's merge) fully unblocked it -- updated registry.ts, AGENTS.md,
  and the corresponding test to match cve-bench/cybergym/owasp's "not stubbed" treatment.
- Updated bountybench's own tests/README/doc comments off the old metadata.detectOnly shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wire owasp's score() to the merged detect-only mode cybergym/owasp score() error messages say capabilities are unmerged, but they merged today

1 participant