Merge pull request #311 from kernel/hypeship/kernel-runtime-upstream

feat(browser-runtime): add native Kernel support

Author: Perry2004

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [0.9.2] - 2026-08-18
 ### Added
+- Added Kernel as a managed remote browser runtime with live view and downloaded replay recordings.
 - Added support for the [WebBrain](https://github.com/webbrain-one/webbrain) harness. Thanks to @alectimison-maker.
 
 ### Fixed

README.md

@@ -296,7 +296,7 @@ Once the container starts, the script prints a **noVNC URL** (e.g. `http://local
 clawbench-batch --models your-model --cases-suite v2 --all-cases
 ```
 
-`your-model` is a key you configured in step 1; `--cases-suite v2` runs the full V2 corpus (swap in `v1-lite` for the 20-task subset). Add `--max-concurrent N` to run tasks in parallel (default 2 locally, 1 with Browserbase) and `--harness <name>` to pick an agent. Each task is intercepted and scored by the `deepseek-v4-pro` judge from step 1 — pass `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings land under `./test-output/`.
+`your-model` is a key you configured in step 1; `--cases-suite v2` runs the full V2 corpus (swap in `v1-lite` for the 20-task subset). Add `--max-concurrent N` to run tasks in parallel (default 2 locally, 1 with Kernel or Browserbase) and `--harness <name>` to pick an agent. Each task is intercepted and scored by the `deepseek-v4-pro` judge from step 1 — pass `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings land under `./test-output/`.
 
 **By hand, to produce a human reference run:**
 
@@ -330,7 +330,7 @@ Full registry: [`src/clawbench/runtime/harnesses/harnesses.yaml`](src/clawbench/
 
 | I want to… | Where |
 | --- | --- |
-| Use a managed remote browser instead of a local container | [`docs/browser-runtimes.md`](docs/browser-runtimes.md) — Browserbase setup, options, recording URLs |
+| Use a managed remote browser instead of a local container | [`docs/browser-runtimes.md`](docs/browser-runtimes.md) — Kernel and Browserbase setup, options, and recordings |
 | Run V2 through the Harbor framework (and run it fast) | [`docs/harbor.md`](docs/harbor.md) — conversion, judge wiring, concurrency, troubleshooting |
 | See every CLI command and flag | [`docs/cli.md`](docs/cli.md) |
 
@@ -754,7 +754,7 @@ Each session records five layers of synchronized data under `/data/`:
 
 | Layer              | File                   | Description                                                     |
 | ------------------ | ---------------------- | --------------------------------------------------------------- |
-| Session replay     | `recording.mp4` or `run-meta.json` recording URL | Local H.264 video or Browserbase Session Inspector replay |
+| Session replay     | `recording.mp4` or `run-meta.json` recording URL | Local/Kernel H.264 video or Browserbase Session Inspector replay |
 | Action screenshots | `screenshots/*.png`    | Throttled timestamped PNGs captured after browser actions       |
 | Browser actions    | `actions.jsonl`        | Every DOM event (click, keydown, input, pageLoad, scroll, etc.) |
 | HTTP traffic       | `requests.jsonl`       | Every HTTP request with headers, body, and query params         |

docs/README.zh-CN.md

@@ -243,7 +243,7 @@ uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4
 clawbench-batch --models your-model --cases-suite v2 --all-cases
 ```
 
-`your-model` 是你在第 1 步里配置的 key；`--cases-suite v2` 跑完整 V2 语料（换成 `v1-lite` 则是 20 题子集）。`--max-concurrent N` 控制并发（本地默认 2，Browserbase 默认 1），`--harness <name>` 选择智能体。每个任务都会被拦截并由第 1 步配置的 `deepseek-v4-pro` judge 打分 —— 加 `--no-judge` 可跳过评分。`batch-summary.json` 和各次运行的录制都会写到 `./test-output/`。
+`your-model` 是你在第 1 步里配置的 key；`--cases-suite v2` 跑完整 V2 语料（换成 `v1-lite` 则是 20 题子集）。`--max-concurrent N` 控制并发（本地默认 2，Kernel 或 Browserbase 默认 1），`--harness <name>` 选择智能体。每个任务都会被拦截并由第 1 步配置的 `deepseek-v4-pro` judge 打分 —— 加 `--no-judge` 可跳过评分。`batch-summary.json` 和各次运行的录制都会写到 `./test-output/`。
 
 **自己上手操作，产出人工参考轨迹：**
 
@@ -277,7 +277,7 @@ harness 是驱动浏览器的智能体框架，和模型是两个独立维度。
 
 | 我想…… | 去哪看 |
 | --- | --- |
-| 用托管的远程浏览器代替本地容器 | [`docs/browser-runtimes.md`](browser-runtimes.md) —— Browserbase 配置、参数、录制地址 |
+| 用托管的远程浏览器代替本地容器 | [`docs/browser-runtimes.md`](browser-runtimes.md) —— Kernel 和 Browserbase 配置、参数与录制 |
 | 用 Harbor 框架跑 V2（并且跑得快） | [`docs/harbor.md`](harbor.md) —— 转换、judge 配置、并发、排错 |
 | 查所有 CLI 命令和参数 | [`docs/cli.md`](cli.md) |
 
@@ -649,7 +649,7 @@ ClawBench 的定位：**真实消费级网站、日常任务、端到端录制**
 
 | 层 | 文件 | 描述 |
 |-------|------|-------------|
-| 会话回放 | `recording.mp4` 或 `run-meta.json` 中的录制 URL | 本地 H.264 视频，或 Browserbase Session Inspector 回放 |
+| 会话回放 | `recording.mp4` 或 `run-meta.json` 中的录制 URL | 本地/Kernel H.264 视频，或 Browserbase Session Inspector 回放 |
 | 动作截图 | `screenshots/*.png` | 浏览器动作后经限流捕获的带时间戳 PNG |
 | 浏览器动作 | `actions.jsonl` | 每个 DOM 事件 (click, keydown, input, pageLoad, scroll 等) |
 | HTTP 流量 | `requests.jsonl` | 每个 HTTP 请求，包含 headers、body 和查询参数 |

docs/browser-runtimes.md

@@ -2,12 +2,42 @@
 
 By default ClawBench launches Chromium inside its own container. You can point it at a managed remote browser instead — useful when the host cannot run containers comfortably, or when you want the provider to handle scaling and session replay.
 
-`--browser-runtime` accepts `local` (default), `browserbase`, `remote-cdp`, and `steel`. **`steel` is reserved and not implemented yet** — selecting it raises an error.
+`--browser-runtime` accepts `local` (default), `kernel`, `browserbase`, `remote-cdp`, and `steel`. **`steel` is reserved and not implemented yet** — selecting it raises an error.
 
 ## Local container (default)
 
 Nothing to configure. Chromium, Xvfb, ffmpeg, noVNC, and the recorder/interceptor all run in the task container; the session video lands at `recording.mp4`.
 
+## Kernel
+
+Put the key in `.env.local`:
+
+```dotenv
+KERNEL_API_KEY=...
+```
+
+Then select the runtime on a single or batch run:
+
+```bash
+uv run clawbench-run test-cases/v1/<case> your-model \
+  --browser-runtime kernel
+
+uv run clawbench-batch --models your-model --all-cases \
+  --browser-runtime kernel
+```
+
+Kernel runs use ClawBench's existing CDP action capture, screenshots, HTTP logging, and request interception. ClawBench starts a Kernel replay with each browser and downloads the completed video to `data/recording.mp4` before deleting the session.
+
+Provider options are passed as JSON. Supported fields are `stealth`, `region`, `proxy`, and `tags`:
+
+```bash
+uv run clawbench-batch --models your-model --all-cases \
+  --browser-runtime kernel \
+  --browser-runtime-options '{"stealth":true,"region":"us-east"}'
+```
+
+Set `KERNEL_BASE_URL` to override the default `https://api.onkernel.com` API endpoint. Batch concurrency defaults to **1**; raise `--max-concurrent` only as far as your Kernel account limit allows.
+
 ## Browserbase
 
 Put the key in `.env.local`:

docs/cli.md

@@ -42,7 +42,7 @@ clawbench-run <case-dir> --human      # human reference run
 | `--output-dir <path>` | `<project>/test-output` | Where run directories are written |
 | `--no-build` | off | Skip building the container image (assumes it exists) |
 | `--no-upload` | off | Skip HuggingFace upload even if `HF_TOKEN` is configured |
-| `--browser-runtime <name>` | `local` | `local`, `browserbase`, `remote-cdp` — see [`browser-runtimes.md`](browser-runtimes.md) |
+| `--browser-runtime <name>` | `local` | `local`, `kernel`, `browserbase`, `remote-cdp` — see [`browser-runtimes.md`](browser-runtimes.md) |
 | `--browser-cdp-url <url>` | — | CDP endpoint for `--browser-runtime remote-cdp` |
 | `--browser-runtime-options <json>` | — | Provider-specific options, e.g. `'{"region":"us-west-2"}'` |
 
@@ -71,7 +71,7 @@ Execution:
 
 | Flag | Default | What it does |
 | --- | --- | --- |
-| `--max-concurrent <n>` | 2 local, 1 Browserbase | Parallel jobs |
+| `--max-concurrent <n>` | 2 local, 1 Kernel/Browserbase | Parallel jobs |
 | `--stagger-delay <s>` | 15 | Minimum seconds between consecutive container starts (rolling start) |
 | `--resume <dir>` | — | Reuse a previous batch's output directory and skip finished runs |
 | `--dry-run` | off | Print the job matrix without running anything |
@@ -125,6 +125,8 @@ See [Reproduce the leaderboard](../README.md#reproduce-the-leaderboard) for the
 | `CONTAINER_ENGINE` | Force `docker` or `podman` |
 | `HF_TOKEN` | Optional upload of runs to HuggingFace |
 | `BROWSERBASE_API_KEY` | Browserbase runtime (from `.env.local`) |
+| `KERNEL_API_KEY` | Kernel runtime (from `.env.local`) |
+| `KERNEL_BASE_URL` | Optional Kernel API base URL override |
 | `CLAWBENCH_JUDGE_*` | Judge credentials for Harbor's verifier — see [`harbor.md`](harbor.md) |
 
 PurelyMail credentials for disposable run emails come from the committed `.env`.

src/README.md

@@ -196,22 +196,22 @@ A stagger delay is applied between job starts since during container startup it
 | `--cases-dir PATH`        | Custom case directory                                     | none                                            |
 | `--all-cases`             | Use all task directories in the selected suite/dir        | false                                            |
 | `--case-range START-END`  | Filter by numeric case ID prefix                          | none                                             |
-| `--max-concurrent N`      | Max parallel jobs; Browserbase defaults to 1               | 2 locally, 1 with Browserbase                    |
+| `--max-concurrent N`      | Max parallel jobs; managed runtimes default to 1            | 2 locally, 1 with Kernel/Browserbase             |
 | `--output-dir PATH`       | Base output directory                                     | `test-output`                                    |
 | `--stagger-delay SECONDS` | Minimum gap between consecutive container starts          | 15                                               |
 | `--dry-run`               | Print job matrix without running                          | false                                            |
 | `--no-upload`             | Skip HuggingFace upload for all runs                      | false                                            |
 | `--harness NAME`          | Harness image to use                                      | `openclaw`                                       |
-| `--browser-runtime NAME`  | Browser runtime (`local`, `remote-cdp`, or `browserbase`)  | `local`                                          |
+| `--browser-runtime NAME`  | Browser runtime (`local`, `remote-cdp`, `kernel`, or `browserbase`) | `local`                                  |
 | `--browser-cdp-url URL`   | CDP endpoint used with `--browser-runtime remote-cdp`      | none                                             |
 | `--browser-runtime-options JSON` | Provider options such as Browserbase region or proxies | none                                          |
 
-Browserbase reads `BROWSERBASE_API_KEY` from `.env.local` or the process
-environment. Its provider-hosted Session Inspector recording URL is stored in
-`browser_runtime.recording_url` in `run-meta.json`; local MP4 recording is
-thus omitted to save space and bandwidth. The signed provider CDP URL is mounted into the runtime
-container through a temporary read-only secret file and redacted from saved
-metadata.
+Kernel reads `KERNEL_API_KEY`; Browserbase reads `BROWSERBASE_API_KEY`. Both
+can be set in `.env.local` or the process environment. Kernel replay video is
+downloaded to `data/recording.mp4`; Browserbase's provider-hosted Session
+Inspector URL is stored in `browser_runtime.recording_url` in `run-meta.json`.
+Signed provider CDP URLs are mounted into the runtime container through a
+temporary read-only secret file and redacted from saved metadata.
 
 Signal handling:
 

src/clawbench/runner/batch.py

@@ -49,6 +49,7 @@ def detect_engine() -> str:
     "claw-eval": "test-cases/claw-eval",
 }
 DEFAULT_CASES_SUITE = "v2"
+MANAGED_BROWSER_RUNTIMES = frozenset({"browserbase", "kernel"})
 
 
 def load_models_yaml() -> dict:
@@ -289,6 +290,8 @@ async def run_job(
                     cmd_parts += ["--harness", harness]
                 if browser_runtime:
                     cmd_parts += ["--browser-runtime", browser_runtime]
+                    if browser_runtime in MANAGED_BROWSER_RUNTIMES:
+                        cmd_parts.append("--hide-browser-viewer")
                 if browser_cdp_url:
                     cmd_parts += ["--browser-cdp-url", browser_cdp_url]
                 if browser_runtime_options:
@@ -590,13 +593,13 @@ async def async_main(args: argparse.Namespace) -> int:
     running_procs.clear()
     browser_runtime = getattr(args, "browser_runtime", None) or "local"
     if getattr(args, "max_concurrent", None) is None:
-        args.max_concurrent = 1 if browser_runtime == "browserbase" else 2
+        args.max_concurrent = 1 if browser_runtime in MANAGED_BROWSER_RUNTIMES else 2
     if (
-        browser_runtime == "browserbase"
+        browser_runtime in MANAGED_BROWSER_RUNTIMES
         and args.harness == "claude-code-chrome-extension"
     ):
         print(
-            "ERROR: browserbase runtime does not support the "
+            f"ERROR: {browser_runtime} runtime does not support the "
             "claude-code-chrome-extension harness"
         )
         return 1
@@ -821,7 +824,7 @@ def main() -> None:
         "--max-concurrent",
         type=int,
         default=None,
-        help="Max parallel jobs (default: 1 for browserbase, otherwise 2)",
+        help="Max parallel jobs (default: 1 for managed runtimes, otherwise 2)",
     )
     p.add_argument("--output-dir", default="test-output", help="Base output directory")
     p.add_argument(
@@ -863,7 +866,7 @@ def main() -> None:
         choices=BROWSER_RUNTIME_CHOICES,
         default=None,
         help=(
-            "Browser runtime provider: local, remote-cdp, steel, or browserbase "
+            "Browser runtime provider: local, remote-cdp, steel, browserbase, or kernel "
             "(default: local)"
         ),
     )

src/clawbench/runner/run.py

@@ -127,7 +127,7 @@ def main():
         choices=BROWSER_RUNTIME_CHOICES,
         default=None,
         help=(
-            "Browser runtime provider: local, remote-cdp, steel, or browserbase "
+            "Browser runtime provider: local, remote-cdp, steel, browserbase, or kernel "
             "(default: CLAWBENCH_BROWSER_RUNTIME or local)"
         ),
     )
@@ -141,6 +141,11 @@ def main():
         default=None,
         help="JSON object with provider-specific browser runtime options",
     )
+    parser.add_argument(
+        "--hide-browser-viewer",
+        action="store_true",
+        help=argparse.SUPPRESS,
+    )
     parser.add_argument(
         "--judge",
         default="deepseek-v4-pro",
@@ -184,11 +189,11 @@ def main():
     if args.human and browser_runtime_provider.name != "local":
         parser.error("human mode currently supports only --browser-runtime local")
     if (
-        browser_runtime_provider.name == "browserbase"
+        browser_runtime_provider.name in {"browserbase", "kernel"}
         and args.harness == "claude-code-chrome-extension"
     ):
         parser.error(
-            "browserbase runtime does not support the "
+            f"{browser_runtime_provider.name} runtime does not support the "
             "claude-code-chrome-extension harness"
         )
 
@@ -230,19 +235,18 @@ def main():
     judge_cfg: dict | None = None
     personal_info_metadata: dict[str, Any] | None = None
     browser_session: BrowserSession | None = None
+    browser_runtime_finalized = False
     browser_runtime_cleaned = False
     browser_cdp_secret_dir: Path | None = None
 
     def _browser_runtime_meta() -> dict[str, Any]:
         if browser_session is not None:
             return browser_session.to_metadata()
         mode = "local" if browser_runtime_provider.name == "local" else "remote"
-        recording_mode = (
-            "x11"
-            if mode == "local"
-            else "provider"
-            if browser_runtime_provider.name == "browserbase"
-            else "disabled"
+        recording_mode = getattr(
+            browser_runtime_provider,
+            "default_recording_mode",
+            "x11" if mode == "local" else "disabled",
         )
         return {
             "provider": browser_runtime_provider.name,
@@ -260,7 +264,10 @@ def _browser_runtime_meta() -> dict[str, Any]:
         }
 
     def _recording_required() -> bool:
-        return browser_session is None or browser_session.recording_mode == "x11"
+        return browser_session is None or browser_session.recording_mode in {
+            "x11",
+            "provider-download",
+        }
 
     def _write_browser_cdp_secret(session: BrowserSession) -> Path | None:
         nonlocal browser_cdp_secret_dir
@@ -293,6 +300,13 @@ def _cleanup_browser_cdp_secret() -> None:
         except OSError:
             pass
 
+    def _finalize_browser_runtime() -> None:
+        nonlocal browser_runtime_finalized
+        if browser_session is None or browser_runtime_finalized:
+            return
+        browser_runtime_provider.finalize(browser_session, output_dir)
+        browser_runtime_finalized = True
+
     def _cleanup_browser_runtime() -> None:
         nonlocal browser_runtime_cleaned
         if browser_session is None or browser_runtime_cleaned:
@@ -527,10 +541,16 @@ def handle_sigint(sig, frame):
                     )
                 console.print("  Open the URL above to watch the agent in real-time.\n")
             elif browser_session.viewer_url:
-                viewer_url = browser_session.to_metadata()["viewer_url"]
-                console.print(
-                    f"\n  Browser viewer: [link={viewer_url}]{viewer_url}[/link]\n"
-                )
+                if args.hide_browser_viewer:
+                    console.print(
+                        f"\n  Remote browser: {browser_session.provider} "
+                        "(viewer URL hidden in batch mode)\n"
+                    )
+                else:
+                    viewer_url = browser_session.viewer_url
+                    console.print(
+                        f"\n  Browser viewer: [link={viewer_url}]{viewer_url}[/link]\n"
+                    )
             else:
                 console.print(
                     f"\n  Remote browser: {browser_session.provider} "
@@ -565,6 +585,12 @@ def handle_sigint(sig, frame):
             model_cfg=None if args.human else model_cfg,
         )
 
+        phase = "finalizing_browser_runtime"
+        _finalize_browser_runtime()
+        phase = "cleaning_browser_runtime"
+        _cleanup_browser_runtime()
+        _cleanup_browser_cdp_secret()
+
         # Stage 2 — LLM judge (default on, --no-judge to skip).
         # Only invoked when stage 1 (intercepted) succeeded; otherwise the
         # task already fails at stage 1 and there's nothing to judge.
@@ -616,10 +642,6 @@ def handle_sigint(sig, frame):
                 )
                 print(f"Judge skipped due to error: {e}")
 
-        phase = "cleaning_browser_runtime"
-        _cleanup_browser_runtime()
-        _cleanup_browser_cdp_secret()
-
         # Write run metadata
         phase = "writing_run_meta"
         duration = time.time() - start_time