feat: add abench method

docs/documentation/how_to_guides.md

@@ -7,3 +7,5 @@
 ###4. How to [implement `Task` chaining](how_tos/task_chaining.md)
 
 ###5. How to [add `Tool` verbosity control support](how_tos/tool_verbosity.md)
+
+###6. How to [run benchmarks concurrently (`abench`)](how_tos/concurrent_benchmarking.md)

docs/documentation/how_tos/concurrent_benchmarking.md

@@ -0,0 +1,150 @@
+# How to Run Benchmarks Concurrently (`abench`)
+
+**Goal**: Run many trials in parallel instead of one at a time, under limits you control.
+
+**When to use this**: You have more than a handful of task/trial pairs (e.g. `pass@k` over many tasks) and each trial spends most of its time waiting on an LLM or a tool. Serial `bench()` runs them one after another; the concurrent path overlaps them and can cut wall-clock time dramatically.
+
+`Corral` exposes concurrency through two twin entry points on `CorralRunner`:
+
+| Entry point | Use it when | Notes |
+|-------------|-------------|-------|
+| `runner.bench(max_concurrency=...)` | You're in **synchronous** code | Internally drives an event loop via `anyio.run`, so it must **not** be called from inside a running event loop. |
+| `await runner.abench(...)` | You're **already inside an event loop** | The async counterpart; pair it with an `AsyncCorralRouter`. |
+
+At `max_concurrency=1` both keep the byte-for-byte serial behaviour, so the serial `bench()` from [Tutorial 1](../tutorials/tut_1_first_benchmark.md) is just the degenerate case of this same API.
+
+## 1. Provide an `agent_factory` (required for concurrency)
+
+A single `BaseAgent` accumulates per-run state (messages, token usage, hooks, harness/SDK sessions), so it **cannot** be shared by two trials running at once. Concurrent benchmarking therefore takes a *factory* — a callable `(TrialContext) -> BaseAgent` — and calls it once per trial, giving every trial its own isolated agent.
+
+```python
+from corral import CorralRunner, TrialContext
+from corral.agents import BaseAgent, ReActAgent
+
+
+def make_agent(context: TrialContext) -> BaseAgent:
+    # A fresh agent per trial. `context` carries task_id, trial_index,
+    # session_id and benchmark_run_id if you want per-trial wiring.
+    return ReActAgent(model="openai/gpt-4o")
+
+
+runner = CorralRunner(interface, agent_factory=make_agent)
+```
+
+You may also pass a shared `agent=` alongside the factory — it's used only for run metadata / report labeling, while the factory mints the agent each trial actually runs.
+
+/// info
+Forgetting the factory fails fast: running any concurrent path with only a shared `agent` raises `Concurrent benchmarking needs a fresh agent per trial`. There's no silent sharing or deep-copying.
+///
+
+## 2. Run it — synchronous
+
+```python
+result = runner.bench(
+    task_ids=["task_1", "task_2", "task_3"],
+    trials_per_task=3,
+    k_values=[1, 2, 3],
+    max_concurrency=5,  # up to 5 trials in flight across the whole run
+    run_name="react_gpt4o",
+)
+
+print(f"Average Score: {result.average_score():.2f}")
+print(f"Pass@1:        {result.pass_at_k(1):.2f}")
+result.generate_report("results.json")
+```
+
+Independent tasks overlap directly; a single collector records every trial and checkpoints deterministically, so resuming an interrupted run skips already-recorded trials.
+
+## 3. Run it — asynchronous
+
+Inside an event loop, use `abench` directly with an `AsyncCorralRouter`:
+
+```python
+import anyio
+from corral import AsyncCorralRouter, CorralRunner, TrialContext
+from corral.agents import BaseAgent, ClaudeCodeAgent
+
+
+async def main() -> None:
+    interface = AsyncCorralRouter("http://localhost:8000")
+
+    def make_agent(_context: TrialContext) -> BaseAgent:
+        return ClaudeCodeAgent(model="claude-sonnet-4-5")
+
+    runner = CorralRunner(interface, agent_factory=make_agent)
+
+    result = await runner.abench(
+        task_ids=await interface.get_available_tasks(),
+        trials_per_task=3,
+        k_values=[1, 2, 3],
+        max_concurrency=5,
+        run_name="claude_code_spectra",
+    )
+    print(f"Overall score: {result.total_score:.3f}")
+
+
+anyio.run(main)
+```
+
+## 4. Overlap repeated trials of the *same* task
+
+By default, repeated trials of one task stay **serialised** (they'd otherwise fight over the same mutable server-side environment). To overlap them too, raise `max_concurrency_per_task`:
+
+```python
+runner.bench(
+    task_ids=["task_1"],
+    trials_per_task=4,
+    max_concurrency=4,
+    max_concurrency_per_task=2,  # two trials of task_1 at once
+)
+```
+
+Values above `1` require the environment server to support **trial runtimes** (isolated per-trial environments); the runner then routes each trial through its own runtime so repeated trials no longer share state. If the server doesn't support it, the run fails fast with a clear error.
+
+**Envs that keep process-global state are isolated (or serialised) automatically.** Trial runtimes isolate per-trial *Python state* (a fresh `CorralState` and workspace), but they can't isolate **process-global** state — module globals or a stateful native library shared by the whole interpreter. Each env declares its safety through `Environment.concurrency`:
+
+- `"thread"` *(default)* — no such state; trials overlap freely in-process.
+- `"process"` — keeps process-global state (e.g. wetlab's reaktoro system lives in module globals). If the environment server was launched with a **process-worker pool** (its `run_server` was given a `build_envs` builder), each such trial runs in its **own worker process**, so independent trials overlap up to the pool size. Without a pool, they fall back to running **one at a time across the whole benchmark**.
+- `"serial"` — the explicit escape hatch: never overlaps anything, even with a worker pool.
+
+You don't set this per run — it ships with the env. It's a correctness guardrail: a stateful env stays correct under a concurrent run, and gets the speed-up once its server runs a worker pool.
+
+> Chained tasks are an exception: a dependency chain pins all of an episode's nodes to one worker (to share their dependency outputs), so `"process"` nodes in a chain stay serialised even with a pool.
+
+## 5. Fine-grained limits with `ConcurrencyConfig`
+
+The two scalars above are shorthands. For the full set of levers, pass a [`ConcurrencyConfig`](../reference.md):
+
+```python
+from corral import ConcurrencyConfig, CorralRunner
+
+config = ConcurrencyConfig(
+    global_trials=16,  # == max_concurrency
+    per_task=4,  # == max_concurrency_per_task
+    per_model={"claude-opus": 8, "gpt-5.6": 16},  # per-model rate-limit cap
+    tool_jobs_per_trial=6,  # background jobs within one trial
+    configure_apps=2,  # simultaneous app-configure steps
+)
+
+# Set it once on the runner...
+runner = CorralRunner(interface, agent_factory=make_agent, concurrency=config)
+
+# ...or pass it per call (this wins over the runner's and over the scalars):
+runner.bench(task_ids=[...], trials_per_task=4, concurrency=config)
+```
+
+| Field | What it bounds |
+|-------|----------------|
+| `global_trials` | Max trials executing simultaneously across the whole benchmark. |
+| `per_task` | Max simultaneous trials of the **same** task (needs trial-runtime support above `1`). |
+| `per_model` | Max simultaneous trials per **agent model** — your provider/rate-limit lever. A model absent from the map is bounded only by `global_trials`. |
+| `tool_jobs_per_trial` | Max background jobs running at once **within a single trial runtime**. Defaults to the server's historical value. |
+| `configure_apps` | Max trials configuring their additional apps/services at once — the fixed-port / expensive-setup lever. |
+
+**Precedence**: a `concurrency=` passed to `bench`/`abench` beats one set on the `CorralRunner`, which beats the scalar `max_concurrency` / `max_concurrency_per_task` arguments. `per_model` and `configure_apps` are enforced only on the concurrent path (they're meaningless when one trial runs at a time).
+
+## 6. Chained tasks
+
+A dependency chain (see [Task Chaining](task_chaining.md)) runs concurrently as an *episode DAG*: each trial round is an isolated episode whose sibling tasks run in parallel, dependents start once their parents succeed, broken chains propagate as *unreachable*, and rounds overlap up to the global limit — all from the same `bench(max_concurrency=...)` / `abench` call.
+
+**Done**: Your benchmark now runs trials in parallel, bounded by exactly the limits you set, with the same results, checkpoints, and reports as the serial path.

docs/documentation/reference.md

@@ -519,6 +519,85 @@ Single source of truth for one environment's runtime state (`corral.backend.stat
 
 ---
 
+## CorralRunner Class
+
+Executes agents against an environment and aggregates results.
+
+**Constructor**:
+
+```python
+runner = CorralRunner(
+    interface,  # CorralRouter or AsyncCorralRouter
+    agent=None,  # a single shared BaseAgent (serial runs)
+    agent_factory=None,  # (TrialContext) -> BaseAgent (required for concurrency)
+    concurrency=None,  # default ConcurrencyConfig for this runner
+    checkpoint_dir="./benchmark_checkpoints",
+    enable_surrender=False,
+    metrics=None,
+)
+```
+
+Exactly one agent source is required. A shared `agent` accumulates per-run state, so it is fine for serial `bench()` but **cannot** back concurrent trials; pass an `agent_factory` for those. Both may be supplied — the factory then mints each trial's agent and the shared agent is used only for run metadata / report labeling.
+
+### `bench(...) -> BenchmarkResult`
+
+Run the benchmark synchronously. Key parameters:
+
+- `task_ids` (list[str] | None): Tasks to run; `None` runs every available task.
+- `trials_per_task` (int): Trials per task (for `pass@k`).
+- `k_values` (int | list[int] | None): k values for pass@k metrics.
+- `run_name` (str | None): Name used in the report filename.
+- `max_concurrency` (int, default `1`): Max trials in flight across the whole run. `1` is the byte-for-byte serial path; `> 1` runs concurrently via `abench` and **requires** an `agent_factory`.
+- `max_concurrency_per_task` (int, default `1`): Max simultaneous trials of the *same* task. Above `1` requires the server to support trial runtimes.
+- `agent_factory` (AgentFactory | None): Per-call override of the runner's factory.
+- `concurrency` (ConcurrencyConfig | None): Full config; when given it supersedes the two scalars above.
+
+When the effective concurrency exceeds `1`, `bench` drives an event loop via `anyio.run`, so it must **not** be called from inside a running event loop.
+
+### `abench(...) -> BenchmarkResult`
+
+Async counterpart of `bench` with the same arguments. Use it directly when you are already inside an event loop (pair it with an `AsyncCorralRouter`).
+
+---
+
+## ConcurrencyConfig Dataclass
+
+Frozen value object bounding how many trials — and how much per-trial work — run at once. Pass it to `CorralRunner(concurrency=...)` or to `bench`/`abench(concurrency=...)`.
+
+**Fields**:
+
+- `global_trials` (int, default `1`): Max trials executing simultaneously across the whole benchmark (the `max_concurrency` scalar).
+
+- `per_task` (int, default `1`): Max simultaneous trials of the same task (the `max_concurrency_per_task` scalar). Values above `1` require server-side trial-runtime support.
+
+- `per_model` (Mapping[str, int] | None): Per-model cap on simultaneous trials, keyed by agent model name (e.g. `{"claude-opus": 8}`). A model absent from the map is bounded only by `global_trials`. Each cap must be `>= 1`.
+
+- `tool_jobs_per_trial` (int, default `4`): Max background jobs running at once within a single trial runtime; forwarded to each runtime's `JobManager`.
+
+- `configure_apps` (int | None): Max trials configuring their additional apps/services simultaneously. Must be `>= 1` when set.
+
+`per_model` and `configure_apps` are enforced only on the concurrent path (`abench`, and `bench` when effective concurrency exceeds `1`). All numeric fields are validated (`>= 1`) at construction.
+
+---
+
+## TrialContext Dataclass
+
+Immutable description of the trial an `AgentFactory` is building for. Passed to the factory so it can bind an agent to one specific trial.
+
+**Fields**:
+
+- `task_id` (str): Task this trial belongs to.
+
+- `trial_index` (int): Zero-based index of this trial within the task.
+
+- `session_id` (str): Benchmark session id (checkpoint namespace).
+
+- `benchmark_run_id` (str): Identifier for the whole benchmark invocation.
+
+`AgentFactory` is the callable protocol `(TrialContext) -> BaseAgent`.
+
+---
+
 ## BenchmarkResult Dataclass
 
 Results from benchmark execution.

docs/documentation/tutorials/tut_1_first_benchmark.md

@@ -183,4 +183,6 @@ Congratulations! You've run your first `Corral` benchmark.
 
 Try modifying the environment to use different numbers or add more tasks.
 
+Once you have several tasks (or run `trials_per_task > 1` for `pass@k`), running them one at a time gets slow. See [How to run benchmarks concurrently (`abench`)](../how_tos/concurrent_benchmarking.md) to overlap trials under limits you control.
+
 ---

pyproject.toml

@@ -40,6 +40,18 @@ dependencies = [
   "pydantic",
   "scikit-learn",
   "requests>=2.32.3",
+  # Async HTTP transport for corral.router.AsyncCorralRouter (one shared
+  # AsyncClient the concurrent scheduler can await without blocking the loop).
+  # Already present transitively via fastapi/openai/anthropic/mcp; pinned here
+  # because it is now imported directly by library code.
+  "httpx>=0.27",
+  # Cross-process/-host tool execution: corral.backend.executors and the
+  # _job_worker subprocess cloudpickle the (tool, arguments) pair so that
+  # @tool-decorated closures survive shipping to a ProcessPool / Modal function.
+  # Imported at module load on the `import corral` path (backend/__init__ ->
+  # env -> jobs -> executors), so it must be a hard dependency. It used to be
+  # pulled in transitively via modal, but modal>=1.x no longer requires it.
+  "cloudpickle>=3.0",
   "urllib3",
   "openai",
   "uvicorn>=0.34.0",

scripts/validate_tool_docstrings.py

@@ -62,6 +62,14 @@
     MAIN_TAGS + WORKFLOW_NESTED_TAGS + RAISES_NESTED_TAGS + ARGS_TAGS + RETURNS_TAGS
 )
 
+# Repo-relative paths whose @tool functions are intentionally exempt from the
+# tagged docstring format. These expose generic, shared tools (e.g. the
+# filesystem helpers) that don't participate in the verbosity system and use
+# plain docstrings instead.
+EXCLUDED_FILES = {
+    "src/corral/utils/io_tools.py",
+}
+
 # Common misspellings / variant spellings that should be flagged
 COMMON_MISSPELLINGS: dict[str, str] = {
     "ARGS_SYNTACTIC": "ARGS_SYNTACTICAL",
@@ -298,6 +306,14 @@ def find_tool_files(repo_root: Path) -> list[Path]:
     return sorted(files)
 
 
+def _is_excluded(filepath: Path, repo_root: Path) -> bool:
+    """Return True if the file is exempt from docstring validation."""
+    resolved = filepath.resolve()
+    return any(
+        resolved == (repo_root / excluded).resolve() for excluded in EXCLUDED_FILES
+    )
+
+
 def validate_file(filepath: Path) -> list[Violation]:
     """Validate all @tool functions in a single file."""
     all_violations: list[Violation] = []
@@ -321,6 +337,9 @@ def main(argv: list[str] | None = None) -> int:
         # Standalone mode: scan all task tool files
         files = find_tool_files(repo_root)
 
+    # Drop files that are intentionally exempt from the tagged format.
+    files = [f for f in files if not _is_excluded(f, repo_root)]
+
     if not files:
         return 0
 

src/corral/__init__.py

@@ -1,17 +1,26 @@
+from corral.concurrency import (
+    AgentFactory,
+    ConcurrencyConfig,
+    TrialContext,
+)
 from corral.report.metrics import (
     Metric,
     MetricMetadata,
     TaskMetric,
     get_default_metrics,
 )
-from corral.router.routes import CorralRouter
+from corral.router.routes import AsyncCorralRouter, CorralRouter
 from corral.run import CorralRunner
 
 __all__ = [
+    "AgentFactory",
+    "AsyncCorralRouter",
+    "ConcurrencyConfig",
     "CorralRouter",
     "CorralRunner",
     "Metric",
     "MetricMetadata",
     "TaskMetric",
+    "TrialContext",
     "get_default_metrics",
 ]