Skip to content

Commit 4825b5d

Browse files
Perry2004Perry2004
authored
Merge pull request #197 from reacher-z/docs/v1-vs-v2-comparison
docs: V1 → V2 comparison table (7 axes, sourced from 6 sub-agent audits)
2 parents 8824188 + bd237b2 commit 4825b5d

1 file changed

Lines changed: 133 additions & 0 deletions

File tree

‎docs/v1-vs-v2.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# ClawBench V1 → V2: What Changed
2+
3+
Side-by-side comparison of the V1 corpus (the original 2026-04 release) and the V2 corpus (the new default). Numbers in the tables below were verified against the corpus files, the judge code, the website, and the public HF datasets — see "Provenance" at the bottom of each section for the exact files we read.
4+
5+
The short version: **V2 is intercept-only, judge-LLM-only, and self-contained — anyone can reproduce a leaderboard row with `clawbench-reproduce --model deepseek-v4-flash` and one OpenRouter key.**
6+
7+
---
8+
9+
## 1. Task corpus
10+
11+
| Axis | V1 | V2 | Δ |
12+
|---|---:|---:|---|
13+
| Tasks (`task.json` count) | 153 | 130 | -23 |
14+
| Unique sites (`metadata.sites_involved`) | 144 | 64 | -80 |
15+
| Sites in both | 44 | 44 | — |
16+
| Tasks reused from V1 (by ID) | — | 51 | 33% of V1 carried over |
17+
| New V2-only tasks | — | 79 | 60% of V2 is fresh |
18+
| Top metaclass | daily-life (21) | daily-life (12) | smoothed |
19+
| Categories | 15 | 19 | +4 (charity, civic-engagement, fitness …) |
20+
21+
Provenance: `test-cases/v1/**/task.json`, `test-cases/v2/**/task.json`, `task.schema.json`.
22+
23+
---
24+
25+
## 2. Safety design (the headline V2 change)
26+
27+
| Safety axis | V1 | V2 | Result |
28+
|---|---|---|---|
29+
| `eval_schema` filled (interceptable) | 70/153 = **46%** | 130/130 = **100%** | V2 makes interception mandatory |
30+
| Tasks that submit a payment / checkout | 29/153 = 19% | 7/130 = 5% | -76% |
31+
| Job-application tasks | 14 | 7 | halved |
32+
| Newsletter / account signups | 14 | 10 | reduced |
33+
| Irreversible-action coverage | 28+ tasks with placeholder schema (could slip past interceptor) | 0 placeholder schemas | **V2 closes the gap** |
34+
| Real-account requirement | none | none | unchanged |
35+
| Personal info | disposable email + dummy `alex_green` profile | same (mailinator pool planned) | unchanged |
36+
37+
V1's "agentic evaluator" model meant tasks without a regex were judged after the fact from the human-reference trace — fine for offline analysis, but the agent could in principle submit a real request before the eval ran. V2 requires every task to declare a URL+method pattern up front, so the interceptor blocks the request *before* it leaves the container.
38+
39+
Provenance: `eval/agentic_eval.md` (V1 nine-rule rubric); `docs/scoring.md`; sample `task.json` files from both corpora.
40+
41+
---
42+
43+
## 3. Evaluation rubric + judge
44+
45+
| Axis | V1 | V2 |
46+
|---|---|---|
47+
| Judge | Claude Code subagent + paired human-reference trace | LLM-only on intercepted HTTP body |
48+
| Inputs | 5-layer trace (actions + requests + screenshots + recording + agent-messages) + human ref ≈ 100 MB | instruction + intercepted URL/method/body, truncated to ≤6 KB |
49+
| Rubric | 9 semantic rules (payment must be attempted, phone wall = PASS, CAPTCHA must attempt, etc.) | Lenient default: *"no explicit contradiction → match"*. Strict opt-in: *"ambiguous → mismatch"* |
50+
| Cost per batch | 16 parallel Claude Code subagents × ~$0.30/task ≈ $50+ | 1 judge call/task × ~$0.01 ≈ $1 |
51+
| Requires Claude Code installed? | yes | no |
52+
| Requires human-reference trace? | yes | no |
53+
| Per-task artifact | `{model}-eval-results.{csv,json}` (per-batch) | `judge.json` (strict) / `judge_llm.json` (lenient) per task + `rescore-summary.json` per batch + `eval_results/<batch>/{per_task.csv,summary.json}` |
54+
55+
V2's lenient rubric is what the public leaderboard uses; the strict rubric is preserved for ablation. Both are publishable Python modules (`src/clawbench/runner/judge.py`, `src/clawbench/runner/judge_llm.py`).
56+
57+
Provenance: `eval/agentic_eval.md`, `src/clawbench/runner/judge.py`, `src/clawbench/runner/judge_llm.py`, `src/clawbench/eval/rescore.py`.
58+
59+
---
60+
61+
## 4. Interceptor / URL-pattern specificity
62+
63+
| Metric | V1 | V2 |
64+
|---|---:|---:|
65+
| `eval_schema` filled | 46% (70/153) | 100% (130/130) |
66+
| Placeholder reliance | 54% (83/153) | 0% |
67+
| HTTP methods covered | POST 100% | POST 92% / GET 8% / PUT <1% |
68+
| Average pattern length | 53 chars | 40 chars |
69+
| Patterns > 60 chars (complex) | 44% of declared patterns | 5% |
70+
71+
V2 patterns are shorter and more diverse because the corpus now covers more "navigate-then-confirm" tasks (GET-able resource fetches) rather than V1's near-uniform "POST a form" pattern. None of the sampled patterns matched known third-party telemetry endpoints (no false-positive intercepts in our spot check).
72+
73+
Sample V2 patterns:
74+
- `myrecipes\.com/api/v\d+/review/save`
75+
- `change\.org/api-proxy/graphql.*op=GenerateAiDraft|.*op=CreatePetition`
76+
- `ravelry\.com/discuss/[^/]+/topics`
77+
78+
Provenance: sampled 20 `task.json` files per corpus.
79+
80+
---
81+
82+
## 5. Identity / personal info
83+
84+
| Aspect | V1 (current) | V2 (current + planned) |
85+
|---|---|---|
86+
| Email service | PurelyMail (`clawbench.cc` domain, paid Anthropic account) | same today; mailinator + 100-name pool planned (Task #14) |
87+
| Test personae | single (`alex_green`) | same today; 100-name pool planned |
88+
| PDF resume | hardcoded name + companies + degrees, runtime email injection | same today; placeholder-only template planned |
89+
| Teacher / referee names | hardcoded in personal-info JSON | dynamic generator planned |
90+
| Setup cost | requires Anthropic PurelyMail account | shared mailinator.com domain (~$900/yr; reproducible by anyone with the credential) |
91+
92+
Identity is the one axis where V1 ≈ V2 today; the V2-plan upgrades land in a follow-up PR once the mailinator domain is procured.
93+
94+
Provenance: `src/clawbench/runtime/shared/alex_green_personal_info.json`, `src/clawbench/utils/resume_template.json`, `src/clawbench/runner/run.py` (PurelyMail integration), `docs/superpowers/specs/2026-05-09-claw-bench-v2-update-design.md` (V2 identity plan).
95+
96+
---
97+
98+
## 6. Public surfaces
99+
100+
| Surface | V1 mentioned | V2 mentioned | V2 default? |
101+
|---|---|---|---|
102+
| arXiv `2604.08523` | yes (153 tasks, 144 sites) | no | n/a (paper pre-dates V2) |
103+
| GitHub README | yes | yes | yes (6-tab leaderboard, V2 Hermes first; 2026-05-20 news entry headlines V2) |
104+
| claw-bench.com | yes (V1 153) | yes (V2 130) | yes (hero strapline = V2; default leaderboard tab = V2 Hermes) |
105+
| HF `NAIL-Group/ClawBench` (task definitions) | yes | yes | tie (both shown) |
106+
| HF `TIGER-Lab/ClawBenchV2Trace` | for context | primary | yes (V2-only repo) |
107+
| HF Space `TIGER-Lab/ClawBench` leaderboard | yes (V1 traces link) | yes (V2 traces link) | yes (`gr.Radio(value="v2")` in `app.py`) |
108+
109+
Only the arXiv abstract still describes V1 alone — that requires a paper v2 revision, which is a separate task from this PR.
110+
111+
---
112+
113+
## 7. Saving the eval config per run (small reliability improvement)
114+
115+
Each `judge.json` / `judge_llm.json` written by `clawbench-rescore` already records the run's effective config — the model that judged it, the rubric used, the rubric prompt (via the file name), and the raw judge reply for audit. The per-batch `rescore-summary.json` re-states these at batch level. Concretely, every V2 judge file contains:
116+
117+
```json
118+
{
119+
"match": true,
120+
"reason": "…",
121+
"judge_model": "deepseek-v4-pro",
122+
"rubric": "lenient",
123+
"raw": "{\"match\": true, …}"
124+
}
125+
```
126+
127+
The `eval_results/<batch>/summary.json` aggregates: `judge_model`, `rubrics`, and per-rubric percentages formatted as `X.X%`. Anyone re-running the same `clawbench-rescore` command against the same `TIGER-Lab/ClawBenchV2Trace` snapshot can compare their numbers to ours field-by-field. This is the spec for "every eval's config is saved" — V2 already complies; V1 did not (V1's `{model}-eval-results.csv` only stored verdict + brief reason).
128+
129+
---
130+
131+
## TL;DR
132+
133+
V2 takes the parts of V1 that depended on human-curated artefacts (human-reference traces, Claude Code subagents, partially-filled regex schemas) and replaces them with three machine-checkable inputs: a complete `eval_schema`, a stateless LLM judge, and a per-run config receipt. The result is a benchmark that anyone can install, run, and audit end-to-end without infrastructure that only the original team has — which is the actual "release V2" claim.

0 commit comments

Comments
 (0)