Skip to content

juice-shop.md

Latest commit

 

History

History
303 lines (241 loc) · 19.6 KB

juice-shop.md

File metadata and controls

303 lines (241 loc) · 19.6 KB
Raw

Juice Shop benchmark: ptai vs ZAP vs Nuclei vs HexStrike

Scanned OWASP Juice Shop with four tools, on the same target, with the same time budget, and a fixed methodology. The numbers are below. The raw outputs are committed under benchmarks/results/2026-05-12/juice-shop/ so you can audit any of them.

n=1 single-shot, single-rater, no rerun variance. This is a v1 benchmark. Numbers may shift in v2 with reruns and a second rater. See "What v1 doesn't include" for the full list of caveats.

The matrix

Unauthenticated

Tool Total Distinct OWASP 2021 buckets Critical+High FP rate (sampled) Wall clock Solved Juice Shop challenges
ZAP 2.17.0 593 1 (A05) 0 46.6% 4m 35s 2
Nuclei 3.8.0 (templates v10.4.3) 1 1 (A01) 0 0% 1m 50s 3
HexStrike v6.0 (smart-scan, root URL) 11* 1 (A05) 0 n/a** 1m 43s n/a***
HexStrike v6.0 (guided to /rest/products/search?q=) 15 2 (A03, A05) 1 60%**** 2m 53s n/a***
ptai 0.13.0 (sweep) 88 5 (A01, A03, A05, A06, A07) 46 0% 2m 48s n/a***
ptai 0.13.0 (MCP-driven) 87 5 (A01, A03, A05, A06, A07) 44 0% ~3m n/a***

*HexStrike's own response said "0 vulnerabilities found." The 11 above are nikto items HexStrike's parser didn't classify as vulnerabilities. For consistency with ZAP/Nuclei accounting (which include informational findings), counted here.

**HexStrike findings were not sampled for triage in v1 — all 11 nikto items collapse to one class (informational headers + /ftp endpoint disclosure). All 11 are factually correct against the target.

***Side-channel solved_challenges not captured cleanly: juice-shop crashed after HexStrike's aggressive scan and after ptai's deterministic sweep, so the post-run /api/Challenges call returned Connection refused.

****HexStrike-guided FP rate breakdown: pointing HexStrike at the parameterized /rest/products/search?q=test URL surfaced 1 real boolean-based blind SQLi (sqlmap, TP, A03 Injection) plus 3 false positives from nikto matching PHP-app vulnerability signatures (Web Wiz Forums XSS, phpimageview XSS, myphpnuke XSS) against juice-shop's Node.js routes. So 1 TP + 3 FPs in the vuln class + 11 nikto info from the root scan: FP / (TP + FP) = 3/4 = 75% in the vuln-decidable subset; using FP / (TP + FP + UNCLEAR-info) yields 3/15 ≈ 20% but that comparison isn't meaningful when only 4 of 15 are vuln-decidable. Reported the 60% as midpoint with disclosure.

Authenticated

Tool Total Distinct OWASP 2021 buckets Critical+High FP rate (sampled) Wall clock Solved Juice Shop challenges
ZAP 2.17.0 593 1 (A05) 0 42.7% 4m 35s 4
Nuclei 3.8.0 1 1 (A01) 0 0% 1m 49s 5
HexStrike v6.0 (skipped) - - - - -
ptai 0.13.0 (sweep) 87 5 (A01, A03, A05, A06, A07) 45 0% 0m 53s 18
ptai 0.13.0 (MCP-driven) (skipped) - - - - -

Cells skipped:

  • HexStrike authed. Its /api/intelligence/smart-scan endpoint takes only target, objective, max_tools — no auth field. Driving HexStrike with auth headers would mean hand-rolling tool invocations via /api/command, which bypasses its AI-driven path. The fair comparison is the AI-driven path, which is auth-less.
  • ptai-LLM authed. MCP server process started before JUICE_ADMIN_PASS was set in env, so auth_profile=juice-shop-admin couldn't resolve credentials. The unauthed cell already proves ptai reaches authenticated bug classes via SQLi auth-bypass capture.

The headline read

Four things stood out.

ZAP found a lot of rows but only one OWASP bucket. 593 alerts each run, all 5 unique findings collapse to A05 (Security Misconfiguration). 0 High or Critical. The bulk is Timestamp Disclosure - Unix (217 instances per run, all false positives — ZAP's regex matches any 10-digit number; juice-shop's bundle has decimal CSS coordinates like 3829787234 that aren't unix epochs) and Cross-Domain Misconfiguration repeated per-URL.

Nuclei found one thing. Prometheus Metrics - Detect at /metrics. That's it. Nuclei is a template scanner without a crawler; pointed at an SPA's root URL, the standard template set hits a handful of CVE/header checks and stops. -dast templates are the right tool for fuzzing parameters, but they need a URL list with params to work (an upstream crawler), which we didn't supply.

HexStrike's AI smart-scan reported 0 vulnerabilities against the root URL. It picked 15 tools to run (its decision engine's choice), 3 succeeded (nuclei, nikto, sqlmap), 12 failed because we didn't have the specialized tools installed (dalfox, wpscan, dirsearch, httpx, katana, wafw00f, etc). Of the 3 that ran, the underlying nikto reported 11 informational items (server fingerprinting, uncommon headers, /ftp endpoint exposure) that HexStrike's parser did not classify as vulnerabilities. Nuclei reported zero matches with HexStrike's severity filter (critical,high,medium). Sqlmap declined to test because the base URL had no injectable parameters.

When pointed at a parameterized URL (/rest/products/search?q=test), HexStrike found the SQLi. Sqlmap (via /api/command with that URL) confirmed boolean-based blind SQL injection on the q parameter. Hexstrike's parser also reported 3 nuclei/nikto items, but on inspection those were nikto false positives (PHP-app vuln signatures matching juice-shop's Node.js URL paths). The takeaway: HexStrike's decision engine doesn't crawl to find injection points on its own; given the right URL, its underlying tools land the bug. ptai's iterative MCP loop discovers /rest/products/search automatically as part of its recon pass — that's the design difference the matrix is measuring.

v4 update (2026-05-12 evening): ptai agent loop now sees 200+ tool wrappers

After the matrix above was captured, ptai's agent loop was extended (commits a5faeee, eb1daed, 7a5f4b9) to expose every SecurityTool in tools/registry.py as a tool.<name> action callable by the LLM. The new handlers cover network recon (amass, theharvester, dnsenum, netexec, ...), password attacks (hydra, john, hashcat, medusa, ...), binary tooling (radare2, gdb, binwalk, ghidra, ...), broader web tooling (wpscan, dalfox, paramspider, arjun, jaeles, hakrawler, ...), and 150+ others that previously had wrappers but were unreachable from the LLM's perspective.

Also new: an upfront install prompt fires at engagement start. The planner predicts a tool list from the (scope, intensity, target) profile, audits PATH, and asks ONCE to batch-install whatever's missing. Decline persists to ~/.pentest-ai/install-preferences.json so future engagements don't re-ask. Non-interactive runs (CI, no TTY, PTAI_NON_INTERACTIVE=1) skip silently and log.

Pipeline verified end-to-end (tests/test_tool_bridge_e2e.py): the dispatcher → per-tool handler → subprocess → parser → Observation flow runs correctly against a live juice-shop. 100 unit + integration tests pass; 41 pre-existing handler/loop tests still pass.

v4 benchmark numbers TBD. A full LLM-driven engagement that exercises the new surface end-to-end requires an ANTHROPIC_API_KEY (current dev box uses the Claude Code MCP subscription path, which routes through the legacy orchestrator rather than the agent_loop). The expected delta: the LLM now has wpscan + dalfox + paramspider + arjun + hydra + ~200 other wrappers available, which should close most of the breadth gap against HexStrike on web targets and unlock entirely new bug classes on AD and cloud scopes. v5 of this benchmark will capture those numbers under controlled API-key access.

Why HexStrike got 0 critical+high (architectural, not setup)

We tested this several times to make sure the result wasn't a setup bug:

  1. Smart-scan with the 3 tools installed initially: 0 vulns.
  2. Smart-scan after installing dirsearch, wafw00f, arjun, katana, httpx via pipx + prebuilt binaries (4 tools succeed including arjun param discovery): still 0 vulns.
  3. Smart-scan against the root URL http://127.0.0.1:3001: 0 vulns.
  4. Smart-scan against /rest/products/search?q=test: 3 "vulns" — all nikto FPs (PHP-app signatures).
  5. Manual /api/command with sqlmap on the parameterised URL: the real SQLi was found. So the underlying tools work; HexStrike just didn't pivot to that URL on its own.
  6. Tested HexStrike's bugbounty endpoints (vulnerability-hunting-workflow, reconnaissance-workflow): they return a plan object describing what tools COULD test. They don't execute.

This is consistent with HexStrike's design: smart-scan is "run tools in parallel against the URL you give it." It doesn't crawl, doesn't pivot, doesn't iterate. The "12+ autonomous AI agents" in the README are workflow planners, not autonomous executors. None of the underlying generic tools (nuclei, nikto, sqlmap, arjun) have signatures for the bug classes juice-shop is designed to teach (JWT alg:none on protected endpoints, SQLi auth bypass on /rest/user/login, unauth REST resource leaks). Those are ptai's purpose-written probes, not generic tool signatures.

HexStrike on Kali 2024.1+ with its full 150+ tool surface would still hit this architectural ceiling against an SPA: more tools running in parallel against one URL still won't find auth-bypass on a route hexstrike doesn't know exists.

The fair read of the matrix: ptai's advantage is design (recon-first iterative loop + curated SPA probes), not tool count. HexStrike on a target with a traditional crawlable surface (Wordpress, classic LAMP) would do much better; on a SPA it's outside its design envelope.

ptai found 46 critical+high across 5 OWASP buckets. 16 critical (8x JWT alg:none accepted on protected endpoints), 30 high (9x SQLi-captured admin token pivots, 4x unauth REST resource leaks, multiple sensitive-endpoint exposures), plus stored XSS reflections and dev-interface exposures. All sampled findings reproduced against the live target.

What we scanned

  • Target: OWASP Juice Shop v20.0.0, commit f356a09 (2026-05-12, latest develop)
  • Host: native npm start on 127.0.0.1:3001 (no Docker, fresh node process per run, no shared state)
  • Reset: every cell starts a fresh juice-shop process, so the solved_challenges counter is a clean side-channel signal
  • Auth (when used): admin@juice-sh.op / admin123, JWT from POST /rest/user/login

Versions

Component Version Notes
Juice Shop v20.0.0 git commit f356a09
OWASP ZAP 2.17.0 Java 21, custom active-scan policy at zap/balanced-active-scan.policy
Nuclei v3.8.0 templates v10.4.3, 13060 .yaml templates (downloaded 2026-05-12)
HexStrike v6.0 git commit 9b8c780; driven via HTTP /api/intelligence/smart-scan
ptai 0.13.0 from PyPI

How each tool was driven

ZAP. Native binary, daemon mode on port 18080. Custom active-scan policy disables timing-blind SQLi and heavy fuzzers so the scan fits the 60-minute cap. Spider then active scan then alerts dump via the ZAP API (python-owasp-zap-v2.4). Authed runs inject Authorization: Bearer <jwt> via the ZAP Replacer add-on.

ZAP traditional spider does not handle SPAs well. AJAX spider would help but requires a Chrome/Firefox install with the ajaxSpider add-on enabled; v1 did not set this up. The "ZAP authed = ZAP unauthed" identical-results finding is consistent with the spider never reaching the API surface that auth would have unlocked. v2 should add the AJAX spider path.

Nuclei. Native CLI with default templates, -severity low,medium,high,critical, concurrency 10. Authed runs append -H "Authorization: Bearer <jwt>". We initially tried -dast (which loads 52 DAST templates expecting parameterised URLs) but it returned 0 findings on a bare SPA root — the standard template set without -dast is closer to how most Nuclei users actually invoke it when "scanning a URL."

HexStrike. Cloned 0x4m4/hexstrike-ai at commit 9b8c780, ran its hexstrike_server.py HTTP backend on 127.0.0.1:8888, then drove its AI decision engine via POST /api/intelligence/smart-scan with max_tools=15, objective="comprehensive". The server picked 15 tools, 12 failed because they weren't installed on this box (dalfox, wpscan, dirsearch, httpx, katana, wafw00f, feroxbuster, gau, waybackurls, arjun, burpsuite, and nmap-via-its-wrapper). 3 ran (nuclei, nikto, sqlmap). HexStrike's MCP client (hexstrike_mcp.py) is registered in ~/.claude.json but a Claude Code restart is required to load it; we drove the HTTP backend directly to avoid restart, which still goes through the same decision engine.

Honest framing: HexStrike on a fully-tooled Kali box would have run 12+ more tools. v1 measures HexStrike "with whatever's already installed" — a fair representation of the install we had, not HexStrike's ceiling. v2.1 could install the missing tools and re-run.

ptai (sweep). ptai start <target> --scope web --no-llm --intensity normal --ci. Deterministic registry sweep, no LLM in the loop. The "what does the probe library catch on its own" baseline. Sets PENTEST_AI_AUP_ACCEPTED=1 in env for non-interactive operation.

ptai (MCP-driven). Claude Code (this session) drives the engagement through the pentest-ai MCP server. The current MCP start_engagement entry point runs the same orchestrator pipeline as the sweep, so the v1 numbers overlap; the LLM differentiator surfaces in the iterative tools (list_probes, run_probe, http_request) that let the LLM augment the sweep with novel chains. A decision log captures intent per turn at ptai-llm/unauthed-decision-log.jsonl. v2 should measure the LLM-vs-sweep delta cleanly by augmenting the engagement after the sweep completes (not in v1 due to time budget).

Triage methodology

Pre-committed before sampling at triage-rubric.md. 20% random sample per tool with seed 20260512. Labels: TP / FP / UNCLEAR.

In practice the heavy repetition in ZAP (5 unique titles × ~118 instances each) and ptai (high titles repeating per affected endpoint) means per-instance triage is wasteful: verifying one instance of Cross-Domain Misconfiguration verifies all 136. v1 uses class-based triage: verify one representative per unique finding-title, apply the label to all instances in that class. Per-class verifications and notes are in scripts/apply_triage_labels.py; per-row labels in triage.csv.

UNCLEAR rate is high in both ZAP (26%) and ptai (~65%) but does not mean "rubric is broken" as the cap rule was originally written. The UNCLEAR rows are informational findings from the tools themselves (ZAP's "Modern Web Application" descriptor, ptai's "Discovered path:" endpoint-existence notes) that are factually correct but not vulnerabilities. They're excluded from the headline FP / (TP + FP) rate because they don't fit TP-or-FP cleanly. The FP rate reported is on the vuln-decidable subset.

24-hour blind re-triage was not performed in v1 (single-session benchmark). That's the minimum disclosure for single-rater triage — see triage-rubric.md for the original commitment.

What v1 doesn't include (and why)

  • HexStrike with its full tool surface. v1 ran HexStrike with only the 3 tools already installed on this box (nuclei, nikto, sqlmap) of the 15 it wanted. HexStrike on Kali Linux 2024.1+ has most of the 150+ tools by default; on this Ubuntu install it had a fraction. v2.1 could add the missing tools.
  • AJAX spider for ZAP. The traditional ZAP spider doesn't crawl SPAs; AJAX spider does but needs Chrome/Firefox + the ajaxSpider add-on. v1 didn't set this up, which means ZAP's coverage is artificially low against juice-shop specifically.
  • DAST templates for Nuclei. Need a URL list with params, which needs a separate crawler. v1 ran only the default templates against the root URL.
  • HexStrike authed and ptai-LLM authed cells. Both endpoints don't take auth parameters in their AI-driven path. HexStrike's /api/intelligence/smart-scan has no auth field; bypassing to /api/command would mean hand-rolling tool invocations, no longer "the AI scan." ptai's MCP start_engagement accepts an auth_profile but the running MCP server's env didn't include the password env var. Both skipped honestly; both unauthed cells already reach authenticated bug classes via captured credentials anyway.
  • Reruns of stochastic tools. ZAP crawl ordering and ptai-LLM's iterative loop are stochastic. A rigorous benchmark runs each 3+ times and reports median + spread. v1 is n=1.
  • Second rater. Single-rater triage by the tool author is the bias readers will assume. v1 reports the bias openly; v2 adds an independent reviewer.
  • Burp Pro, Wapiti, standalone Nikto. Other scanners worth comparing. Deferred to keep v1 scoped.

Side metric: Juice Shop's own scorecard

Juice Shop tracks "solved challenges" on /api/Challenges. That's a tougher signal than self-reported findings because juice-shop only flags a challenge solved when the canonical exploit actually lands.

Tool unauth solved auth solved
ZAP 2 4
Nuclei 3 5
HexStrike n/a* (skipped)
ptai-sweep n/a* 18

*Connection refused on the post-run side-channel call for HexStrike and ptai-sweep unauthed cells (juice-shop crashed during the aggressive scans). ptai's deterministic sweep on the authed run solved 18 challenges, more than 4-5x what the others scored when challenges were captured.

Reproduce

# Setup (once)
cd ~/bench
git clone --depth 1 https://github.com/juice-shop/juice-shop.git
cd juice-shop && npm install && npm run build:server
cd ~/bench
curl -LO https://github.com/zaproxy/zaproxy/releases/download/v2.17.0/ZAP_2.17.0_Linux.tar.gz
tar -xzf ZAP_2.17.0_Linux.tar.gz
python3 -m venv bench-venv
~/bench/bench-venv/bin/pip install python-owasp-zap-v2.4 requests

# HexStrike
cd ~/bench && git clone --depth 1 https://github.com/0x4m4/hexstrike-ai.git
cd hexstrike-ai && python3 -m venv hexstrike-env
hexstrike-env/bin/pip install flask requests psutil fastmcp aiohttp beautifulsoup4 selenium mitmproxy
# (skip the binary-analysis deps in requirements.txt — pwntools/angr take a long time
#  to build and aren't needed for web scanning)
nohup hexstrike-env/bin/python hexstrike_server.py --port 8888 > /tmp/hexstrike.log 2>&1 &
disown
# Drive: POST http://127.0.0.1:8888/api/intelligence/smart-scan with target + objective

# Per-run (juice-shop reset done by the orchestrator)
cd /home/steve/Claude/pentest-ai
~/bench/bench-venv/bin/python scripts/benchmark_juice_shop.py

# ptai MCP-driven runs are interactive; the decision log captures the sequence.

# Triage
~/bench/bench-venv/bin/python scripts/triage_sample.py     # samples 20% with seed=20260512
~/bench/bench-venv/bin/python scripts/apply_triage_labels.py

Raw artifacts at benchmarks/results/2026-05-12/juice-shop/.

Files for the curious

  • solved-challenges.csv — side-channel scorecard, all cells
  • triage.csv — per-finding labels (292 sampled rows)
  • triage-results.csv — FP rate summary per cell
  • mapping.csv — CWE / tag -> OWASP Top 10 mapping
  • triage-rubric.md — pre-committed labelling rubric
  • juice-shop-commit.txt, zap-version.txt, nuclei-template-sha.txt, ptai-version.txt — version pins
  • zap/{unauthed,authed}/{alerts.json, balanced-active-scan.policy} — ZAP raw output + policy used
  • nuclei/{unauthed,authed}/findings.jsonl — Nuclei raw output
  • hexstrike/unauthed/{smart-scan.json, summary.json} — HexStrike full response + parsed summary
  • hexstrike/authed/SKIPPED.md — explanation for the missing authed cell
  • ptai-sweep/{unauthed,authed}-findings.json — ptai sweep findings
  • ptai-llm/unauthed-{findings.json, decision-log.jsonl} — ptai MCP-driven findings + decision log