Add worker_timeout, label and env to eval suite manifests

changelog/5712.added.2.md

@@ -0,0 +1 @@
+- A suite entry can carry a `label:` and an `env:`. The label names the entry in the dashboard, the failure and pass-rate summaries, `results.jsonl` (a new `"label"` field; `"bot"` still holds the path) and the artifact filenames, where it is made filesystem-safe, and `-p/--pattern` matches it as well as the bot path; `env:` is added to the spawned bot's environment over the suite's own. Together they let one bot file run under several configurations (a model, a reasoning effort) as separate entries whose logs and recordings do not overwrite each other. `EvalRun` gains `label` (default: the bot path) and `env`.

changelog/5712.added.md

@@ -0,0 +1 @@
+- `pipecat eval suite` manifests take a `worker_timeout:` (seconds), overridden by `-w/--worker-timeout`, capping how long a run's harness worker may take before the suite kills it and reports the run as an error. It used to be fixed at 600 s, which a long scripted scenario could legitimately exceed, losing the whole run. Left unset, the cap is now derived from the scenario: a scripted scenario's turn budgets summed (each turn's largest `within_ms`, or the default timeout), or a simulation's `max_duration_s`, with a floor of 600 s and 60 s on top for the judge and teardown.

changelog/5712.changed.md

@@ -0,0 +1 @@
+- `PipelineWorker`'s default `idle_timeout_frames` now include `LLMFullResponseStartFrame`, `LLMTextFrame` and `TTSTextFrame`, so a pipeline streaming an LLM response counts as active. The default only reset on speaking and transcription frames, none of which flow in a text-only conversation, so a healthy text-mode bot (an eval driven with `send-text`, a chat bot) was cancelled by the 300 s idle timeout mid-conversation. `PipelineTask` shares the default.

scripts/release-evals/README.md

@@ -98,25 +98,82 @@ flags:
 uv run python -m pipecat.evals suite -d manifest.yaml [-p PATTERN] [-s SCENARIO] [-c N] [-n NAME] [-t SECS] [-a] [--no-cache] [--repeat N]
 ```
 
-Each run writes to `test-runs/<name>/` (a timestamp when `-n` is omitted):
+Each run writes to `test-runs/<name>/` (a timestamp when `-n` is omitted). The
+`<label>` in the filenames is the entry's `label:` (its bot path unless set — see
+[One bot, several configurations](#one-bot-several-configurations)) made
+filesystem-safe: `voice/voice-cartesia.py` becomes `voice_voice-cartesia.py`.
 
-- `logs/<bot>__<scenario>.log` — the bot subprocess output.
-- `logs/<bot>__<scenario>.eval.log` — the harness's decision trace (always
+- `logs/<label>__<scenario>.log` — the bot subprocess output.
+- `logs/<label>__<scenario>.eval.log` — the harness's decision trace (always
   written; invaluable for diagnosing a flake).
-- `logs/<bot>__<scenario>.debug.log` — the harness's full per-pipeline logs
+- `logs/<label>__<scenario>.debug.log` — the harness's full per-pipeline logs
   (user speech / bot speech transcription / judge / harness), one section per
   pipeline. Written whenever `-d/--debug` is passed, which `run.sh` always does.
-- `recordings/<bot>__<scenario>.wav` — the conversation audio for audio-mode
+- `recordings/<label>__<scenario>.wav` — the conversation audio for audio-mode
   scenarios. The manifest sets `record: true`, so these are produced by default;
   pass `-a/--audio` to force recording on if a manifest has it off.
 
 Useful flags: `-c/--concurrency`, `-t/--timeout` (default per-expectation
-timeout in seconds, for expectations without their own `within_ms`), and
-`--no-cache` (re-synthesize user audio every turn instead of reusing the cache).
+timeout in seconds, for expectations without their own `within_ms`),
+`-w/--worker-timeout` (seconds a run's harness worker may take before the suite
+kills it and reports an error; by default derived from the scenario, see below),
+and `--no-cache` (re-synthesize user audio every turn instead of reusing the cache).
 Everything in the manifest header except the `suite:` list can also be overridden
 on the command line (the command line wins) — `--bots-dir`, `--scenarios-dir`,
-`--runs-dir`, `--base-port`, `--cache-dir`, `--spawn`, `--python` — so a manifest
-can be just a `suite:` list with the rest supplied as flags.
+`--runs-dir`, `--base-port`, `--cache-dir`, `--spawn`, `--python`,
+`--worker-timeout` — so a manifest can be just a `suite:` list with the rest
+supplied as flags.
+
+### Manifest keys
+
+The header (everything but `suite:`):
+
+| Key              | Default        | Meaning                                                                                  |
+| ---------------- | -------------- | ---------------------------------------------------------------------------------------- |
+| `bots_dir`       | `.`            | Bot paths in `suite:` are relative to it (itself relative to the manifest).              |
+| `scenarios_dir`  | `scenarios`    | A bare scenario name resolves to `<scenarios_dir>/<name>.yaml`.                          |
+| `runs_dir`       | `eval-runs`    | Where `<name>/logs/` and `<name>/recordings/` go.                                        |
+| `spawn`          | `{python} {bot} -t eval --port {port}` | How a bot is started; `{port}` is assigned per run.              |
+| `python`         | the suite's    | The interpreter `{python}` stands for.                                                   |
+| `concurrency`    | `4`            | Runs at a time.                                                                          |
+| `repeat`         | `1`            | Attempts per (bot, scenario); see [Measuring flakiness](#measuring-flakiness).           |
+| `base_port`      | `7900`         | First port; each run gets the next one.                                                  |
+| `record`         | `false`        | Record conversation audio (audio-mode runs only).                                        |
+| `cache_dir`      | user cache     | Where synthesized user audio is cached.                                                  |
+| `worker_timeout` | derived        | Seconds a run's harness worker may take before it is killed and the run reported as an error. Unset, the cap is the scenario's own budget with a 600 s floor, plus 60 s: a scripted scenario's turn budgets summed (each turn's largest `within_ms`, or `-t/--timeout` when none is set), or a simulation's `max_duration_s`. Set it when a run is legitimately longer than the harness can tell from the file, or shorter to fail a wedged run faster. |
+
+A `suite:` entry:
+
+| Key           | Required | Meaning                                                                                     |
+| ------------- | -------- | ------------------------------------------------------------------------------------------- |
+| `bot`         | yes      | The bot file, relative to `bots_dir`.                                                       |
+| `scenarios`   | yes      | Scenario names (`scripted/<name>`, `simulated/<name>`) or `.yaml` paths relative to the manifest. |
+| `label`       | no       | Display name, in the dashboard, the failure summary, `results.jsonl` (`"label"`; `"bot"` stays the path) and the artifact filenames. Default: the bot path. `-p/--pattern` matches it as well as the path. |
+| `env`         | no       | Environment variables added to the spawned bot's, over the suite's own.                    |
+| `runner_body` | no       | A JSON file passed to the bot as `--runner-body`; see [Vision](#vision-image-input).         |
+
+### One bot, several configurations
+
+A benchmark often runs one bot file under several configurations — a model, a
+reasoning effort, a sampling setting — that the bot reads from its environment.
+Rather than a file per configuration, list the entry once per configuration with
+a `label:` telling them apart and the `env:` that selects it:
+
+```yaml
+suite:
+  - bot: voice/voice-anthropic.py
+    label: claude (low effort)
+    env: {ANTHROPIC_EFFORT: low}
+    scenarios: [scripted/multi_turn]
+  - bot: voice/voice-anthropic.py
+    label: claude (high effort)
+    env: {ANTHROPIC_EFFORT: high}
+    scenarios: [scripted/multi_turn]
+```
+
+Each is its own row in the dashboard and its own line in `results.jsonl`, and
+their logs are `claude_low_effort__multi_turn.log` and
+`claude_high_effort__multi_turn.log` rather than one overwriting the other.
 
 ### Measuring flakiness
 
@@ -348,7 +405,8 @@ which prints the conversation as it happens.
 
 ## Adding coverage
 
-- New bot: add an entry to `manifest.yaml` (`bot:` + the `scenarios:` it should run).
+- New bot: add an entry to `manifest.yaml` (`bot:` + the `scenarios:` it should run;
+  a `label:` and `env:` when the same file runs under more than one configuration).
 - New behavior to test: add a `scenarios/scripted/<name>.yaml` and reference it from the
   manifest as `scripted/<name>`.
 - New goal to reach: add a `scenarios/simulated/<name>.yaml` with a `persona:` and reference

src/pipecat/cli/commands/eval.py

@@ -613,11 +613,15 @@ def _row_seconds(group: list[EvalRun]) -> float | None:
 
 
 def _group_key(r: EvalRun) -> tuple[str, str]:
-    return (r.bot, r.scenario)
+    return (r.label, r.scenario)
 
 
 def _grouped_runs(runs: list[EvalRun]) -> dict[tuple[str, str], list[EvalRun]]:
-    """Runs bucketed by (bot, scenario), in first-seen order."""
+    """Runs bucketed by (label, scenario), in first-seen order.
+
+    The label rather than the bot path, so one bot file listed under two
+    labels (two configurations) is two rows.
+    """
     groups: dict[tuple[str, str], list[EvalRun]] = {}
     for r in runs:
         groups.setdefault(_group_key(r), []).append(r)
@@ -662,7 +666,7 @@ def __rich_console__(self, console: Console, options: ConsoleOptions) -> RenderR
             cells.append(
                 (
                     _eval_status_cell(r, self._spinner),
-                    Text(r.bot),
+                    Text(r.label),
                     Text(r.scenario, style="cyan"),
                     Text(detail, style="dim"),
                 )
@@ -851,7 +855,7 @@ def _print_eval_line(r: EvalRun, *, show_attempt: bool = False) -> None:
     if tally:
         extra = f"{tally} {extra}"
     scenario = f"{r.scenario} #{r.attempt}" if show_attempt else r.scenario
-    print(f"  {_color(glyph, code)} {r.bot} {_color(scenario, '36')} {_dim(extra)}", flush=True)
+    print(f"  {_color(glyph, code)} {r.label} {_color(scenario, '36')} {_dim(extra)}", flush=True)
 
 
 def _plural(count: int, noun: str) -> str:
@@ -980,7 +984,7 @@ def _print_failures(failed: list[EvalRun], total: int, *, show_attempt: bool) ->
     for r in failed:
         attempt = f" {_dim('#' + str(r.attempt))}" if show_attempt else ""
         tally = _turn_tally(r)
-        header = f"  {_red('✗')} {r.bot} {_color(r.scenario, '36')}{attempt}"
+        header = f"  {_red('✗')} {r.label} {_color(r.scenario, '36')}{attempt}"
         if tally and isinstance(r.result, EvalScriptResult):
             header = f"{header} {_dim(tally + ' passed')}"
         if r.error:
@@ -1135,7 +1139,7 @@ def suite(
         ..., help="Manifest YAML listing bots + their scenarios (scripted, or simulations)."
     ),
     pattern: str = typer.Option(
-        None, "-p", "--pattern", help="Only bots whose path contains this."
+        None, "-p", "--pattern", help="Only bots whose path or label contains this."
     ),
     scenario: str = typer.Option(None, "-s", "--scenario", help="Only this scenario name."),
     kind: EvalKind = typer.Option(None, "-k", "--kind", help="Only scenarios of this kind."),
@@ -1178,6 +1182,14 @@ def suite(
         help="Default per-expectation timeout in seconds (for expectations without their own "
         "within_ms).",
     ),
+    worker_timeout: float = typer.Option(
+        None,
+        "-w",
+        "--worker-timeout",
+        help="Override manifest worker_timeout: seconds a run's harness worker may take "
+        "before it is killed. Default: derived from each scenario's turn budgets or "
+        "max_duration_s, with a 600s floor, plus 60s.",
+    ),
     spawn: str = typer.Option(None, "--spawn", help="Override manifest spawn template."),
     python: str = typer.Option(None, "--python", help="Override manifest python interpreter."),
     audio: bool = typer.Option(False, "-a", "--audio", help="Record conversation audio."),
@@ -1208,6 +1220,7 @@ def suite(
         base_port=base_port,
         record=audio or None,
         cache_dir=cache_dir,
+        worker_timeout=worker_timeout,
     )
 
     suite = EvalSuite(manifest)