Document live-sweep operational quirks: PITHOS cold-cache clone, Daytona tunnel bridging, timeout sizing (#44)

Author: alanzabihi

AGENTS.md

@@ -43,6 +43,44 @@ Do not implement scientific benchmark bodies until the dependency row is satisfi
 - **pithos:** `{ "type": "pithos", "provider": "<pi-provider-id>", "sandboxMode": "docker" | "local", "maxFindings": <n> }` — requires the `pithos` CLI on `PATH` (`uv tool install git+https://github.com/superagent-ai/PITHOS.git`); parses `TRIAGE.json` + `verify/runtime-summary.json`, not stdout. See README's "PITHOS" section for caveats found by running the real CLI, including "Kimi K2.6 on Azure" for the `src/contenders/pithosKimiAzureExtension.ts` Pi extension (`--provider azure-openai-responses --model kimi-k2.6`) that gets PITHOS onto the same Kimi Azure deployment AutoBrin uses.
 - **command:** `{ "type": "command", "id": "<name>", "command": "<tool> run {repo} --model {model}" }` — stdout may be JSON `ContenderClaim`.
 
+## Running live comparison sweeps
+
+Learned running a real 29-task autobrin-vs-pithos sweep across all four benchmarks. `bench run`'s
+CLI doesn't support constructing a `pithos` contender or setting `contributors`/`maxCycles` — for a
+real multi-task, multi-contender sweep, call `runSingle()` directly from a small script instead of
+shelling out to the CLI.
+
+- **PITHOS's own `git clone --branch <ref>` fails on a fresh cache with a raw commit SHA** (only
+  branches/tags work with `--branch`; a raw SHA 128-fails with `Remote branch <sha> not found`).
+  This is silent and dangerous: PITHOS exits in ~1s with an empty claim, which can coincidentally
+  score as "correct" on a not-vulnerable task purely by luck, masking the failure entirely. Once
+  `.cache/pithos-repos/<owner-repo>/` already has a full clone, subsequent runs against the same
+  repo succeed (PITHOS falls back to fetch+checkout on a warm cache, which does accept a raw SHA).
+  **Always pre-seed the cache with a plain `git clone` for every repo PITHOS will touch before a
+  sweep**, and treat any PITHOS result with `durationS` under a few seconds as suspect, not a
+  genuine miss.
+- **CVE-Bench and BountyBench's Exploit lane, run via `transport: "daytona"`, need the target
+  bridged to the sandbox.** `standUpTarget()` binds the Docker target to `127.0.0.1` on the
+  orchestrator host; the Daytona sandbox is remote and can't reach host-loopback addresses.
+  Bridge with a temporary tunnel (e.g. `cloudflared tunnel --url http://127.0.0.1:<port>`) for the
+  app URL and any other host-loopback field the webapp payload carries (e.g. CVE-Bench's
+  `proofUploadingUrl`), rewrite those fields on the `TargetHandle` before calling
+  `contender.run()`, and tear the tunnel down after. The announced tunnel URL is not immediately
+  reachable — wait for it to actually respond (DNS/edge propagation took ~3-6s in practice) before
+  handing it to the engagement, or the contributor's first request will fail with `ENOTFOUND`.
+- **Size timeouts from real observed data, not the "detect-only is fast" assumption alone.**
+  `detectOnly` bounds evaluation depth (stops after stage 4), not the lead/contributor phase before
+  it — a single hypothesis occasionally ran past 20 minutes without completing cycle 1 under shared
+  model-provider load, well before `detectOnly` had a chance to shorten anything. PITHOS's own full
+  pipeline typically took 20-40 minutes per task, occasionally more under contention. A full
+  (non-detect-only) CyberGym engagement can self-test-confirm a real crash and then spend 45+ more
+  minutes in evaluation (prior_art → dedup → docs_intent → adversarial → exploitation → triage)
+  before writing `evaluate.json` — a too-tight external timeout kills the process and silently
+  discards an already-confirmed finding, indistinguishable after the fact from a genuine miss.
+  Prefer budgeting CyberGym/full-pipeline timeouts north of 90 minutes.
+- If a stored `GH_TOKEN` 403s cloning a private repo (autobrin-flue, or inside a Daytona sandbox),
+  it may be scoped wrong — try `gh auth token` instead of trusting a secrets-file value blindly.
+
 ## Git workflow
 
 - Default branch: `main`.