merge: main into feat/llm-judge

Resolves README News section conflict by keeping the 2026-05-20 V2-default
entry (HEAD) plus main's 2026-05-16 through 2026-05-11 entries. Also fixes
the 2026-05-11 V2Trace link to TIGER-Lab/ClawBenchV2Trace.

batch.py auto-merged: V2 default + claw-eval restored together.
pyproject.toml auto-merged: 0.3.3 + rescore/reproduce scripts.

Author: reacher-z

.github/workflows/run-test.yml

@@ -0,0 +1,46 @@
+name: run-test
+
+on:
+  pull_request:
+    branches: [ main ]
+    paths:
+      - ".github/workflows/run-test.yml"
+      - "src/clawbench/**"
+      - "tests/**"
+      - "pyproject.toml"
+      - "uv.lock"
+  push:
+    branches: [ main ]
+    paths:
+      - ".github/workflows/run-test.yml"
+      - "src/clawbench/**"
+      - "tests/**"
+      - "pyproject.toml"
+      - "uv.lock"
+  workflow_dispatch:
+
+jobs:
+  run-test:
+    name: pytest (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.1.0
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+
+      - name: Run host-side pytest suite
+        run: uv run --frozen pytest

.github/workflows/static-check.yml

@@ -46,6 +46,9 @@ jobs:
       - name: Ruff format check
         run: uv run --frozen ruff format --check .
 
+      - name: Pyright check
+        run: uv run --frozen pyright src/clawbench tests
+
       - name: Build package
         env:
           UV_FROZEN: "true"

.pre-commit-config.yaml

@@ -5,3 +5,10 @@ repos:
       - id: ruff-check
       - id: ruff-format
         args: [--check]
+  - repo: local
+    hooks:
+      - id: pyright
+        name: pyright
+        entry: uv run --frozen pyright src/clawbench tests
+        language: system
+        pass_filenames: false

AGENTS.md

@@ -4,7 +4,7 @@ This file is for coding agents (Claude Code, Cursor, Copilot, etc.) to understan
 
 ## What This Is
 
-ClawBench is a benchmarking framework for evaluating AI web agents on real-world online tasks. V1 lives in `test-cases/v1/` with 153 tasks spanning 144 live websites and 15 life categories; V2 lives in `test-cases/v2/` with 130 tasks; Lite lives in `test-cases/v1-lite/` with 20 link-backed V1 tasks. Each task runs in an isolated Docker container with Chromium, a recording Chrome extension, and an AI agent harness (`openclaw`, `opencode`, `claude-code`, `claude-code-chrome-extension`, `codex`, `browser-use`, `claw-code`, or `hermes`, selectable via `--harness`). The framework captures five layers of data: session replay (MP4), action screenshots, HTTP traffic, browser actions, and agent messages.
+ClawBench is a benchmarking framework for evaluating AI web agents on real-world online tasks. V1 lives in `test-cases/v1/` with 153 tasks spanning 144 live websites and 15 life categories; V2 lives in `test-cases/v2/` with 130 tasks; Lite lives in `test-cases/v1-lite/` with 20 link-backed V1 tasks. Each task runs in an isolated Docker container with  a browser, a recording Chrome extension, and an AI agent harness (`openclaw`, `opencode`, `claude-code`, `claude-code-chrome-extension`, `codex`, `browser-use`, `claw-code`, `hermes`, or `pi`, selectable via `--harness`). The framework captures five layers of data: session replay (MP4), action screenshots, HTTP traffic, browser actions, and agent messages.
 
 ## Project Structure
 
@@ -19,8 +19,8 @@ ClawBench/
       runner/
         run.py                    # Single test-case runner
         batch.py                  # Batch runner (model x case cross-product)
+        run_support/              # Runner-private helpers for single runs
       utils/
-        generate_resume_pdf.py    # Resume PDF generator
         hf_upload.py              # Optional HuggingFace upload helpers
       runtime/
         shared/
@@ -125,7 +125,12 @@ test-output/<model>/<harness>-<case>-<model>-<timestamp>/
                             #   `result`, `state`, `metadata` fields;
                             #   hermes → live `session_meta`/`thinking`/`reasoning`/
                             #   `tool_use`/`tool_result` capture, falling back to
-                            #   Hermes session export when it contains messages)
+                            #   Hermes session export when it contains messages;
+                            #   pi → filtered Pi JSON mode events with
+                            #   `message_start`/`message_end`,
+                            #   `tool_execution_*`, tool-call content, and
+                            #   thinking blocks; streaming `message_update`
+                            #   fragments are omitted)
     screenshots/            # Timestamped PNGs
     recording.mp4           # Full session video
     interception.json       # Interception result

CHANGELOG.md

@@ -6,6 +6,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [0.3.3] - 2026-05-19
+### Changed
+- More data are stored in the `run-meta.json` for better post-hoc analysis and reproducibility, including the hash of the configs, runtime info, and flags used.
+
+### Fixed
+- Fixed several compatibility issues on Windows platforms.
+
+## [0.3.2] - 2026-05-15
+### Added
+- Added the logic to remove the `.log` files from the generated `data/` directory to remove noise.
+- Added the handling to allow models with no visual capabilities to use the `claude-code-browser-extension` harness by skipping the screenshot steps.
+- Added retry logic to the `claude-code-browser-extension` harness to handle temporary rate limits.
+
+## [0.3.1] - 2026-05-13
+### Fixed
+- Removed v1-799 and v2-795 tasks since the current interception schemas have risk of leaking the agent's final action to the end server, which leads to unexpected disturbance of the end business.
+
+## [0.3.0] - 2026-05-09
+### Added
+- Added support for the **pi** harness — [Pi coding agent](https://github.com/earendil-works/pi/tree/main/packages/coding-agent) + [`pi-browser-harness`](https://github.com/amankumarsingh77/pi-browser-harness)
+
 ## [0.2.2] - 2026-05-08
 ### Changed
 - Migrated the PyPI package to `clawbench-eval` instead of `clawbenchmark`.

NOTICE

@@ -1,12 +1,94 @@
 ClawBench
 Copyright 2026 ClawBench Authors
 
-This product includes the following third-party software:
+This product includes the following third-party open source software and
+materials. Transitive package dependencies keep their upstream licenses.
 
-noVNC - Copyright (C) 2024 The noVNC Authors
-Licensed under the Mozilla Public License 2.0 (MPL 2.0)
+Source materials:
+
+Claw-Eval converted task subset
+Source: https://github.com/claw-eval/claw-eval
+Dataset: https://huggingface.co/datasets/claw-eval/Claw-Eval
+Licensed under the MIT License
+
+Direct Python packages:
+
+fpdf2 2.8.7 - Licensed under LGPL-3.0-only
+https://py-pdf.github.io/fpdf2/
+
+huggingface-hub 1.12.0 - Licensed under Apache-2.0
+https://github.com/huggingface/huggingface_hub
+
+PyYAML 6.0.3 - Licensed under the MIT License
+https://github.com/yaml/pyyaml
+
+questionary 2.1.1 - Licensed under the MIT License
+https://github.com/tmbo/questionary
+
+rich 15.0.0 - Licensed under the MIT License
+https://github.com/Textualize/rich
+
+fastapi 0.136.1 - Licensed under the MIT License
+https://github.com/fastapi/fastapi
+
+uvicorn 0.46.0 - Licensed under BSD-3-Clause
+https://uvicorn.dev/
+
+websocket-client 1.9.0 - Licensed under Apache-2.0
+https://github.com/websocket-client/websocket-client/
+
+jsonschema 4.26.0 - Licensed under the MIT License
+https://github.com/python-jsonschema/jsonschema
+
+pre-commit 4.6.0 - Licensed under the MIT License
+https://github.com/pre-commit/pre-commit
+
+ruff 0.15.12 - Licensed under the MIT License
+https://github.com/astral-sh/ruff
+
+Harness and runtime packages:
+
+browser-use 0.12.6 - Licensed under the MIT License
+https://github.com/browser-use/browser-use
+
+LiteLLM 1.83.9 - Licensed under the MIT License
+https://github.com/BerriAI/litellm
+
+websockets 13.1 - Licensed under BSD-3-Clause
+https://github.com/python-websockets/websockets
+
+Hermes Agent, commit bf196a3fc0fd1f79353369e8732051db275c6276 - Licensed under the MIT License
+https://github.com/NousResearch/hermes-agent
+
+agent-browser 0.26.0 - Licensed under Apache-2.0
+https://github.com/vercel-labs/agent-browser
+
+Microsoft Playwright MCP 0.0.70 - Licensed under Apache-2.0
+https://github.com/microsoft/playwright-mcp
+
+opencode-ai 1.4.4 - Licensed under the MIT License
+https://github.com/sst/opencode
+
+OpenClaw 2026.3.13 - Licensed under the MIT License
+https://github.com/openclaw/openclaw
+
+OpenAI Codex CLI 0.120.0 - Licensed under Apache-2.0
+https://github.com/openai/codex
+
+claw-code, commit 00d0eb61d4cfc1abd345dc76023dd21e420f6488 - License NOASSERTION; no license file found at pinned commit
+https://github.com/ultraworkers/claw-code
+
+pi-coding-agent 0.74.0 - Licensed under the MIT License
+https://github.com/earendil-works/pi-mono
+
+pi-browser-harness 0.4.0 - Licensed under the MIT License
+https://github.com/amankumarsingh77/pi-browser-harness
+
+typebox 1.1.38 - Licensed under the MIT License
+https://github.com/sinclairzx81/typebox
+
+noVNC 1.6.0 - Licensed under MPL-2.0
 https://github.com/novnc/noVNC
 
-websockify - Copyright (C) 2024 The websockify Authors
-Licensed under the GNU Lesser General Public License v3.0 (LGPL 3.0)
+websockify 0.13.0 - Licensed under LGPL-3.0
 https://github.com/novnc/websockify