swebench: add SWE-Bench Verified benchmark

.gitignore

@@ -31,6 +31,18 @@ pnpm-lock.yaml
 .gocache
 !.github/scripts/run-go-tests.sh
 .DS_Store
+swebench/config/models/*.local.yaml
+swebench/config/environments/*.local.yaml
+swebench/results/runs/
+swebench/results/artifacts/
+swebench/results/**/*.log
+swebench/results/**/*.traj.json
+swebench/results/**/patches/
+swebench/results/**/traces/
+swebench/results/**/workspaces/
+swebench/data/cache/
+swebench/data/raw/
+swebench/data/generated/
 .gomodcache
 next-env.d.ts
 .codebuddy

swebench/README.md

@@ -0,0 +1,179 @@
+# SWE-Bench Verified
+
+SWE-Bench Verified evaluates software-engineering agents on real GitHub issues.
+This benchmark provides a reproducible path to run the official 500-case test
+split, verify patches with the official local harness, and compare
+mini-SWE-agent against a Go-native `trpc-agent-go` implementation.
+
+## Results
+
+MiniMax M2.5 is used to validate the evaluator and baseline chain against a
+public SWE-Bench reference point. Completed repeat runs are listed separately.
+
+| Model | Agent | Resolved Rate |
+| --- | --- | ---: |
+| MiniMax M2.5 | mini-SWE-agent 2.1.0 | 76.00% |
+| GLM-5.2-Internal | mini-SWE-agent 2.1.0 | 76.40%, 76.60%, 78.80% |
+
+Structured summaries are stored in
+[`results/baseline-mini-swe-agent-m2.5.json`](results/baseline-mini-swe-agent-m2.5.json)
+and
+[`results/experiments/mini-swe-agent-glm-5.2-internal-r3.json`](results/experiments/mini-swe-agent-glm-5.2-internal-r3.json).
+
+The full comparison report will be published in
+[`results/REPORT.md`](results/REPORT.md) and
+[`results/REPORT.zh_CN.md`](results/REPORT.zh_CN.md) after the native
+`trpc-agent-go` run is complete.
+
+## Repository Layout
+
+```text
+swebench/
+  config/                  # Shared local config templates.
+    models/                # Model endpoint templates and ignored local config.
+    environments/          # mini-SWE-agent runtime environment config.
+  data/                    # Fixed case lists and generated dataset metadata.
+  evaluator/               # Shared dataset, verifier, importer, and report CLI.
+  mini-swe-agent-impl/     # mini-SWE-agent baseline adapter.
+  trpc-agent-go-impl/      # Go-native SWE agent implementation.
+  results/                 # Reports and small structured summaries.
+```
+
+## Dataset
+
+| Item | Value |
+| --- | --- |
+| Dataset | `princeton-nlp/SWE-bench_Verified` |
+| Split | `test` |
+| Cases | 500 |
+| Case list | `data/case-lists/verified-test-500.case_ids.txt` |
+| Case list SHA256 | `a6b0fd7c8c2969a0eef892e032250adcfa6d32362d395c246930e61b575ac9b9` |
+
+`data/` only contains lightweight metadata. It must not contain gold patches,
+test patches, hidden test lists, cloned repositories, Docker image caches, or
+raw dataset dumps.
+
+## Quick Start
+
+Run the commands from the benchmark repository root.
+
+### 1. Prepare the evaluator environment
+
+Use a Linux machine with Docker, Go 1.21+, and Python 3.11+.
+
+```bash
+python3.11 -m venv swebench/results/runs/.venv
+source swebench/results/runs/.venv/bin/activate
+
+pip install -U pip
+
+mkdir -p swebench/results/runs/repos
+git clone https://github.com/SWE-bench/SWE-bench.git swebench/results/runs/repos/SWE-bench
+pip install -e swebench/results/runs/repos/SWE-bench
+```
+
+Keep this virtual environment activated for the remaining commands in this
+quick start.
+
+For the mini-SWE-agent baseline runner, see
+[`mini-swe-agent-impl/README.md`](mini-swe-agent-impl/README.md).
+
+### 2. Configure model access
+
+```bash
+cp swebench/config/models/glm-5.2.yaml.example swebench/config/models/glm-5.2.local.yaml
+```
+
+Fill in the endpoint, API key, and required gateway headers in the local YAML.
+Local model config is ignored by git.
+
+### 3. Check evaluator and model access
+
+```bash
+cd swebench
+
+go run ./evaluator doctor \
+  --run-id swebench-doctor \
+  --output results/runs/doctor \
+  --model-config config/models/glm-5.2.local.yaml
+```
+
+The command prints a concise `ok/fail` summary and writes the full details to
+`results/runs/doctor/doctor.json`.
+
+For a healthy evaluator setup, `doctor` should report `ok` for Python,
+SWE-Bench, Docker, dataset loading, managed httpbin, and model smoke checks.
+The mini-SWE-agent check is also expected to be `ok` after installing the
+baseline runner from `mini-swe-agent-impl/README.md`.
+
+### 4. Download dataset
+
+```bash
+go run ./evaluator prepare-data --python python
+```
+
+This downloads SWE-Bench Verified if needed, checks it against the committed
+500-case list, and writes generated metadata files under `data/generated/`.
+
+### 5. Choose an implementation and produce predictions
+
+Choose an implementation, run it, and keep its SWE-Bench predictions file:
+
+- mini-SWE-agent baseline:
+  [`mini-swe-agent-impl/README.md`](mini-swe-agent-impl/README.md)
+- Go-native `trpc-agent-go` agent:
+  [`trpc-agent-go-impl/README.md`](trpc-agent-go-impl/README.md)
+
+The next steps assume `<path-to-preds.json>` points to that file.
+
+### 6. Verify predictions
+
+After an implementation produces SWE-Bench predictions, verify them with the
+official local harness:
+
+```bash
+go run ./evaluator verify \
+  --run-id <run-id> \
+  --target <baseline-or-native> \
+  --predictions <path-to-preds.json> \
+  --output results/runs/<run-id>/local-harness-report/<baseline-or-native> \
+  --harness-workers 1
+```
+
+For subset predictions, `verify` restricts the official harness to the
+prediction instance ids by default.
+
+### 7. Normalize results
+
+```bash
+go run ./evaluator import \
+  --target <baseline-or-native> \
+  --cases data/generated/cases.jsonl \
+  --predictions <path-to-preds.json> \
+  --raw-dir <path-to-raw-run-dir> \
+  --harness-report <path-to-harness-report.json> \
+  --output results/runs/<run-id>/imported
+```
+
+This converts predictions and verifier output into the common case-level result
+format used by reports.
+
+### 8. Write run config
+
+For a single runner manifest:
+
+```bash
+go run ./evaluator run-config \
+  --run-id <run-id> \
+  --target <baseline-or-native> \
+  --cases-manifest data/generated/cases.manifest.json \
+  --runner-manifest <path-to-runner-manifest.json> \
+  --verifier-manifest results/runs/<run-id>/local-harness-report/<baseline-or-native>/verifier_manifest.json \
+  --import-summary results/runs/<run-id>/imported/summary/<baseline-or-native>.json \
+  --harness-report <path-to-harness-report.json> \
+  --model-name <model-name> \
+  --output results/runs/<run-id>/run_config.json
+```
+
+For a sharded mini-SWE-agent full run, use `--shards-manifest` instead of
+`--runner-manifest`.

swebench/README.zh_CN.md

@@ -0,0 +1,174 @@
+# SWE-Bench Verified
+
+SWE-Bench Verified 使用真实 GitHub issue 评估软件工程 Agent 的修复能力。
+本 benchmark 提供一条可复现路径：运行官方 500 case test split，使用官方
+local harness 验证 patch，并对比 mini-SWE-agent 与 Go-native
+`trpc-agent-go` 实现。
+
+## 结果
+
+MiniMax M2.5 用于对齐 SWE-Bench 官网公开参考结果，验证 evaluator 和 baseline
+链路本身可信。已完成的重复实验单独列出。
+
+| Model | Agent | Resolved Rate |
+| --- | --- | ---: |
+| MiniMax M2.5 | mini-SWE-agent 2.1.0 | 76.00% |
+| GLM-5.2-Internal | mini-SWE-agent 2.1.0 | 76.40%, 76.60%, 78.80% |
+
+结构化摘要见
+[`results/baseline-mini-swe-agent-m2.5.json`](results/baseline-mini-swe-agent-m2.5.json)
+和
+[`results/experiments/mini-swe-agent-glm-5.2-internal-r3.json`](results/experiments/mini-swe-agent-glm-5.2-internal-r3.json)。
+
+完整对比报告将在原生 `trpc-agent-go` run 完成后发布到
+[`results/REPORT.md`](results/REPORT.md) 和
+[`results/REPORT.zh_CN.md`](results/REPORT.zh_CN.md)。
+
+## 仓库结构
+
+```text
+swebench/
+  config/                  # 共享本地配置模板。
+    models/                # 模型 endpoint 模板和 ignored 本地配置。
+    environments/          # mini-SWE-agent 运行环境配置。
+  data/                    # 固定 case list 和生成的数据集元信息。
+  evaluator/               # 共享 dataset、verifier、importer 和 report CLI。
+  mini-swe-agent-impl/     # mini-SWE-agent baseline adapter。
+  trpc-agent-go-impl/      # Go-native SWE Agent 实现。
+  results/                 # 报告和小型结构化摘要。
+```
+
+## 数据集
+
+| 项目 | 值 |
+| --- | --- |
+| Dataset | `princeton-nlp/SWE-bench_Verified` |
+| Split | `test` |
+| Cases | 500 |
+| Case list | `data/case-lists/verified-test-500.case_ids.txt` |
+| Case list SHA256 | `a6b0fd7c8c2969a0eef892e032250adcfa6d32362d395c246930e61b575ac9b9` |
+
+`data/` 只保存轻量元信息，不应包含 gold patch、test patch、隐藏测试列表、
+克隆仓库、Docker image cache 或原始数据集 dump。
+
+## Quick Start
+
+以下命令从 benchmark 仓库根目录执行。
+
+### 1. 准备 evaluator 环境
+
+使用安装了 Docker、Go 1.21+ 和 Python 3.11+ 的 Linux 机器。
+
+```bash
+python3.11 -m venv swebench/results/runs/.venv
+source swebench/results/runs/.venv/bin/activate
+
+pip install -U pip
+
+mkdir -p swebench/results/runs/repos
+git clone https://github.com/SWE-bench/SWE-bench.git swebench/results/runs/repos/SWE-bench
+pip install -e swebench/results/runs/repos/SWE-bench
+```
+
+后续 quick start 命令默认继续使用这个已激活的虚拟环境。
+
+mini-SWE-agent baseline runner 的安装和运行见
+[`mini-swe-agent-impl/README.md`](mini-swe-agent-impl/README.md)。
+
+### 2. 配置模型访问
+
+```bash
+cp swebench/config/models/glm-5.2.yaml.example swebench/config/models/glm-5.2.local.yaml
+```
+
+在本地 YAML 中填写 endpoint、API key 和必要的网关 header。本地模型配置已
+被 gitignore。
+
+### 3. 检查 evaluator 和模型访问
+
+```bash
+cd swebench
+
+go run ./evaluator doctor \
+  --run-id swebench-doctor \
+  --output results/runs/doctor \
+  --model-config config/models/glm-5.2.local.yaml
+```
+
+命令会在终端输出简洁的 `ok/fail` 摘要，并将完整细节写入
+`results/runs/doctor/doctor.json`。
+
+健康的 evaluator 环境下，`doctor` 应在 Python、SWE-Bench、Docker、数据集
+加载、managed httpbin 和模型 smoke 检查上返回 `ok`。按
+`mini-swe-agent-impl/README.md` 安装 baseline runner 后，mini-SWE-agent
+检查也应返回 `ok`。
+
+### 4. 下载数据
+
+```bash
+go run ./evaluator prepare-data --python python
+```
+
+该命令会按需下载 SWE-Bench Verified，校验它是否匹配仓库提交的 500 case
+列表，并在 `data/generated/` 下写出生成的元信息文件。
+
+### 5. 选择实现并产出 predictions
+
+选择一个实现运行，并保留 SWE-Bench predictions 文件：
+
+- mini-SWE-agent baseline:
+  [`mini-swe-agent-impl/README.md`](mini-swe-agent-impl/README.md)
+- Go-native `trpc-agent-go` agent:
+  [`trpc-agent-go-impl/README.md`](trpc-agent-go-impl/README.md)
+
+后续步骤中的 `<path-to-preds.json>` 指向该文件。
+
+### 6. 验证 predictions
+
+某个实现产出 SWE-Bench predictions 后，使用官方 local harness 验证：
+
+```bash
+go run ./evaluator verify \
+  --run-id <run-id> \
+  --target <baseline-or-native> \
+  --predictions <path-to-preds.json> \
+  --output results/runs/<run-id>/local-harness-report/<baseline-or-native> \
+  --harness-workers 1
+```
+
+对于 subset predictions，`verify` 默认会将官方 harness 限定到 predictions
+中的 instance ids。
+
+### 7. 整理结果
+
+```bash
+go run ./evaluator import \
+  --target <baseline-or-native> \
+  --cases data/generated/cases.jsonl \
+  --predictions <path-to-preds.json> \
+  --raw-dir <path-to-raw-run-dir> \
+  --harness-report <path-to-harness-report.json> \
+  --output results/runs/<run-id>/imported
+```
+
+这一步会把 predictions 和 verifier 输出转换成报告使用的逐 case 统一结果。
+
+### 8. 写入 run config
+
+对于单个 runner manifest：
+
+```bash
+go run ./evaluator run-config \
+  --run-id <run-id> \
+  --target <baseline-or-native> \
+  --cases-manifest data/generated/cases.manifest.json \
+  --runner-manifest <path-to-runner-manifest.json> \
+  --verifier-manifest results/runs/<run-id>/local-harness-report/<baseline-or-native>/verifier_manifest.json \
+  --import-summary results/runs/<run-id>/imported/summary/<baseline-or-native>.json \
+  --harness-report <path-to-harness-report.json> \
+  --model-name <model-name> \
+  --output results/runs/<run-id>/run_config.json
+```
+
+对于 sharded mini-SWE-agent full run，使用 `--shards-manifest` 替代
+`--runner-manifest`。

swebench/config/README.md

@@ -0,0 +1,18 @@
+# SWE-Bench Config
+
+Shared benchmark configuration templates live here.
+
+## Model Config
+
+```bash
+cp swebench/config/models/glm-5.2.yaml.example swebench/config/models/glm-5.2.local.yaml
+```
+
+Fill in the endpoint, API key, and required gateway headers in the local YAML.
+Local config files are ignored by git and must not be committed.
+
+## Environment Config
+
+`environments/swebench-testbed.yaml` is committed and contains runtime settings
+passed into SWE-Bench containers by mini-SWE-agent. It activates the official
+`testbed` conda environment and limits common native-library thread counts.