Add cross-spec suites (dow suite) for the check x model x domain x temp matrix

`dow aggregate` runs a project's N-way aggregators over a cohort within a SINGLE
spec. A robustness sweep, though, is often a matrix where each cell is its own
spec. `dow suite` aggregates versions drawn from SEVERAL specs at once, declared
in a manifest `specs/<name>.suite.yaml` (its `specs:` list plus the project's own
`evaluation.aggregators`/`plots`). dow still ships none of the coefficients or the
plotting library - the suite only wires in the project's callables.

- spec.py: SuiteSpec dataclass (load/from_dict/to_dict/fingerprint; kind: suite).
- evaluators.py: CohortContext.specs - a parallel per-member spec-name list so an
  aggregator can bucket by spec/model/domain/temperature (empty for single-spec).
- service.py: load_suite/list_suites/select_suite (all|latest|<tag>)/aggregate_suite
  /suite_aggregations/get_suite_aggregation; spec_files() excludes *.suite.yaml so a
  manifest never trips single-spec resolution; composite spec:version member ids.
- store.py: _suites/<name>/ bundle namespace under index["suites"] (kept out of
  list_specs); save/list/get_suite_aggregation guard the suite name with
  _safe_component on read AND write, and use the atomic writers.
- cli.py: `dow suite` command (+ fix the CLI-local _spec_files() to also exclude
  *.suite.yaml, so commit/history/compare/... auto-resolution ignores manifests).
- report.py: print_aggregation is suite-aware (title + N-specs header).
- mcp_server.py: dow_suite tool (server now exposes 15 tools).
- docs/suite.txt (+ regenerated man/dow.1); README/PYPI_README/PROJECT_PLAN.
- tests: test_suite.py (11 - cross-spec select all/latest/tag, bucket-by-spec,
  persistence, manifest-not-a-spec, error capture, empty/missing-agg guards, CLI
  resolution) + a suite-name traversal test + dow_suite in the MCP tool set.

Bumps dow to 2.1.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: theflyingrahul

CHANGELOG.md

@@ -4,6 +4,47 @@ All notable changes to dow are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/), and dow adheres to
 [Semantic Versioning](https://semver.org/).
 
+## [2.1.0] - 2026-07-11
+
+### Added
+- **Cross-spec suites (`dow suite`) — aggregate versions across *several* specs
+  (the check × model × domain × temperature matrix).** `dow aggregate` runs a
+  project's N-way aggregators over a cohort within a *single* spec, but a
+  robustness sweep is often a matrix where each cell is its own spec. A suite is
+  declared in a manifest, `specs/<name>.suite.yaml`, listing the participating
+  `specs:` plus the project's own `evaluation.aggregators`/`plots`:
+
+  ```yaml
+  name: robustness_matrix
+  specs: [check_llama, check_qwen, check_mistral]
+  select: all            # all | latest | <tag>
+  evaluation:
+    aggregators: [suite_metrics.py:agg_matrix]
+    plots: [suite_plots.py:plot_matrix]
+  ```
+
+  `select` chooses the cohort: `all` (every version of each listed spec — the
+  full matrix), `latest` (each spec's newest version), or a tag name (each
+  spec's versions carrying that tag). Each member keeps its own captured config,
+  and the `CohortContext` gains a parallel `specs` list naming each member's
+  spec, so an aggregator can bucket by spec / model / domain / temperature.
+  Member ids are composite `spec:version`. Suite runs persist as durable,
+  git-tracked bundles under `.dow/aggregations/_suites/<name>/` — a **separate
+  namespace** from single-spec aggregations — so a manifest never leaks into
+  `dow history` or single-spec resolution. `dow suite --list`, `--show <id>`,
+  and `--plot` mirror `dow aggregate`. The `.suite.yaml` suffix keeps manifests
+  out of the inference-spec code paths (`spec_files` in both the service core and
+  the CLI now exclude it). dow still ships **none** of the coefficients or the
+  plotting library — the suite only wires in the project's callables.
+- **`dow_suite` MCP tool** (the server now exposes **15** tools), plus the
+  `suite` command in the generated man page and `dow help suite`.
+
+### Security
+- `Store.save_suite_aggregation` / `get_suite_aggregation` validate the suite
+  name (and aggregation id) with `_safe_component` on both the write and read
+  paths, matching the 2.0.4 store-write hardening; regression test added
+  (`tests/test_security.py::test_store_writes_block_traversal_suite_name`).
+
 ## [2.0.4] - 2026-07-11
 
 ### Security

PROJECT_PLAN.md

@@ -326,6 +326,22 @@ An aggregator receives a `CohortContext` whose `members` is one `EvalContext` pe
 
 `dow aggregate` selects the cohort — an explicit list of versions, every version carrying a `--tag`, or the whole history — runs the aggregators, and **persists the result as a durable, git-tracked bundle** under `.dow/aggregations/` (recording the member ids, their fingerprints, the coefficient values, any figures, and a timestamp). A robustness check therefore becomes a citable, reproducible object; `dow aggregate --list` and `--show <id>` retrieve past bundles. Aggregator failures are captured and reported, never fatal.
 
+### Cross-spec suites (the matrix)
+
+`dow aggregate` operates within a **single** spec. A robustness sweep, though, is frequently a **matrix** — the same check run across several models, domains, or temperatures, where each cell of the matrix is its own spec. **`dow suite`** aggregates versions drawn from **several** specs at once, so the whole matrix collapses into one citable object. The participating specs and the project's own aggregators/plots are declared in a manifest, `specs/<name>.suite.yaml` (a distinct suffix that keeps suites out of the inference-spec code paths, so a manifest never leaks into `dow history` or single-spec resolution):
+
+```yaml
+# specs/robustness_matrix.suite.yaml
+name: robustness_matrix
+specs: [check_llama, check_qwen, check_mistral]   # specs to draw versions from
+select: all              # all | latest | <tag>
+evaluation:
+  aggregators: [suite_metrics.py:agg_matrix]      # your callables; dow ships none
+  plots: [suite_plots.py:plot_matrix]
+```
+
+Every member keeps its own captured `config`, and the `CohortContext` gains a parallel **`specs`** list naming each member's spec, so a suite aggregator can bucket by spec / model / domain / temperature. Member ids are composite `spec:version`. `select` chooses the cohort: `all` (every version of each listed spec — the full matrix), `latest` (each spec's newest version), or a tag name (each spec's versions carrying that tag; specs with no match are skipped, and an entirely empty selection is a clear error). Suite runs persist as durable, git-tracked bundles under `.dow/aggregations/_suites/<name>/` — a **separate namespace** from single-spec aggregations — and `dow suite --list`, `--show <id>`, and `--plot` mirror `dow aggregate`. The suite reuses the same aggregator/plot contract, so **dow still ships none of the coefficients or plotting code**.
+
 ### Pluggable plots (figures as content-addressed artifacts)
 
 dow can turn any of these results into a figure without shipping a plotting library. The project references its own plot functions under `evaluation.plots`; each receives a `PlotContext` carrying the analysis `results` to render and a dow-provided `out_dir` to write into, and returns the path(s) of the figure file(s) it produced:
@@ -355,6 +371,7 @@ The commands are task-oriented, not version-control plumbing. Versioning is auto
 | `dow tag <label> [version]` | Tag a version with a free-form label: good, golden, baseline, bad, ... |
 | `dow eval [version]` | Run custom evaluators; compare scores against the previous and last-good versions |
 | `dow aggregate [VERSIONS]` | Aggregate reliability metrics over a cohort of versions (N-way); persist a git-tracked bundle, optionally with figures (`--plot`) |
+| `dow suite [NAME]` | Aggregate versions across several specs (a cross-spec matrix declared in `specs/<name>.suite.yaml`); persist a git-tracked bundle, optionally with figures (`--plot`) |
 | `dow tree` | Visualize evolution: a vertical trunk with branches, as a terminal tree or an exported Mermaid `gitGraph` |
 
 Versions are referred to by simple names (`v1`, `v2`), the shortcuts `last` and `prev`, or any label applied with `dow tag` (for example `good`). They form a tree: every commit records its parent, and `dow commit --from v1` starts a new branch from an earlier version. Command output is rendered in the terminal.
@@ -410,7 +427,7 @@ The trunk `v1 -> v2` is the main line; `v3-branch` forks from v1 and continues t
 
 ### Programmatic and MCP surface
 
-The command line is one front end over a headless core (`dow/service.py`); an AI agent gets the same workflow over the Model Context Protocol. The `dow-mcp` server (install with the `mcp` extra) exposes fourteen tools mirroring the commands above - scaffold, read/write a spec, commit, compare, explain, eval, aggregate, tag, history, inspect, tree, and docs - plus read-only resources (`dow://overview`, `dow://docs/<command>`, `dow://specs`, `dow://spec/<name>`) an agent can attach as context. Because both surfaces call the same core, they never drift apart, and both are data-structure agnostic: when a spec sets `embedding_model: none`, the built-in drift, stability, and verdict come back null (`driftEnabled` is false) and the configuration diff plus the project's own metrics carry the analysis. dow itself still ships no metric, statistic, or plotting code.
+The command line is one front end over a headless core (`dow/service.py`); an AI agent gets the same workflow over the Model Context Protocol. The `dow-mcp` server (install with the `mcp` extra) exposes fifteen tools mirroring the commands above - scaffold, read/write a spec, commit, compare, explain, eval, aggregate, aggregate across specs (suite), tag, history, inspect, tree, and docs - plus read-only resources (`dow://overview`, `dow://docs/<command>`, `dow://specs`, `dow://spec/<name>`) an agent can attach as context. Because both surfaces call the same core, they never drift apart, and both are data-structure agnostic: when a spec sets `embedding_model: none`, the built-in drift, stability, and verdict come back null (`driftEnabled` is false) and the configuration diff plus the project's own metrics carry the analysis. dow itself still ships no metric, statistic, or plotting code.
 
 ---
 
@@ -513,7 +530,7 @@ The tool runs entirely offline by default (mock provider and a built-in hashing
 - [ ] `dow explain` attributes a behavioral change to the configuration difference and flags confounded comparisons.
 - [ ] `dow tree` visualizes the version evolution (terminal tree and exported Mermaid), with branches via `dow commit --from`.
 - [ ] Users can plug in custom evaluators (`evaluation.metrics`); `dow eval` runs them lazily, saves the scores with the version, and compares against the previous and last-good versions.
-- [ ] Users can plug in paired comparators (`evaluation.comparators`), N-way cohort aggregators (`evaluation.aggregators`) surfaced by `dow aggregate` as durable git-tracked bundles, and plot functions (`evaluation.plots`) whose figures dow stores as content-addressed artifacts — dow shipping none of the coefficients or plotting code.
+- [ ] Users can plug in paired comparators (`evaluation.comparators`), N-way cohort aggregators (`evaluation.aggregators`) surfaced by `dow aggregate` as durable git-tracked bundles, cross-spec suites (`specs/<name>.suite.yaml`) surfaced by `dow suite`, and plot functions (`evaluation.plots`) whose figures dow stores as content-addressed artifacts — dow shipping none of the coefficients or plotting code.
 - [ ] `dow tag` applies free-form labels (good, golden, baseline, ...) that are usable as version references.
 - [ ] The tool runs end to end offline through mock or local mode.
 - [ ] A rehearsed two-minute demonstration that lands the closing statement.

PYPI_README.md

@@ -160,6 +160,31 @@ version carrying a `--tag`, or all of them), runs the aggregators, and saves a d
 git-tracked bundle under `.dow/aggregations/`; `dow aggregate --list` and `--show <id>`
 retrieve past results.
 
+### Cross-spec suites (the matrix)
+
+`dow aggregate` works within a single spec. A robustness sweep often spans a whole
+matrix - the same check across several models, domains, or temperatures, where each cell
+is its own spec. `dow suite` aggregates versions drawn from several specs at once. You
+declare the participating specs and the project's own aggregators/plots in a manifest,
+`specs/<name>.suite.yaml`:
+
+```yaml
+# specs/robustness_matrix.suite.yaml
+name: robustness_matrix
+specs: [check_llama, check_qwen, check_mistral]
+select: all              # all | latest | <tag>
+evaluation:
+  aggregators: [suite_metrics.py:agg_matrix]      # your callables; dow ships none
+  plots: [suite_plots.py:plot_matrix]
+```
+
+Each member keeps its own captured `config` (and the `CohortContext` carries a parallel
+`specs` list), so your aggregator can bucket by spec / model / domain / temperature.
+Member ids are composite `spec:version`. `select` chooses the cohort: `all` (the full
+matrix), `latest` (each spec's newest version), or a tag name. Runs persist as durable,
+git-tracked suite bundles, separate from single-spec aggregations, so they never leak
+into `dow history`; `dow suite --list`, `--show <id>`, and `--plot` work as for aggregate.
+
 ### Pluggable plots
 
 dow can render results to figures without shipping a plotting library: reference your
@@ -172,10 +197,10 @@ evaluation:
     - plots.py:forest_plot          # your matplotlib (or any) code; dow ships none
 ```
 
-Run `dow compare --plot` or `dow aggregate --plot`. dow copies each figure into the
-content-addressed artifact store (`.dow/artifacts/`, git-ignored) and records its hash
-and size; for an aggregation the figure is referenced from the persisted bundle, so it
-stays regenerable while the bytes stay out of git.
+Run `dow compare --plot`, `dow aggregate --plot`, or `dow suite --plot`. dow copies each
+figure into the content-addressed artifact store (`.dow/artifacts/`, git-ignored) and
+records its hash and size; for an aggregation or suite the figure is referenced from the
+persisted bundle, so it stays regenerable while the bytes stay out of git.
 
 ### Non-text outputs
 
@@ -224,10 +249,10 @@ should operate on:
 }
 ```
 
-The 14 tools mirror the CLI: `dow_list_specs`, `dow_init`, `dow_read_spec`,
+The 15 tools mirror the CLI: `dow_list_specs`, `dow_init`, `dow_read_spec`,
 `dow_write_spec`, `dow_commit`, `dow_compare`, `dow_explain`, `dow_eval`,
-`dow_aggregate`, `dow_history`, `dow_inspect`, `dow_tag`, `dow_tree`, and
-`dow_docs`. They return structured JSON (config diffs, metrics, comparator and
+`dow_aggregate`, `dow_suite`, `dow_history`, `dow_inspect`, `dow_tag`, `dow_tree`,
+and `dow_docs`. They return structured JSON (config diffs, metrics, comparator and
 aggregator results, the version tree, and Mermaid - plus, for text outputs, drift
 scores and verdicts), so a client can run the full edit, commit, compare loop.
 The server also exposes read-only resources an MCP client can attach as context:

README.md

@@ -183,6 +183,33 @@ version carrying a `--tag`, or all of them), runs the aggregators, and saves a d
 git-tracked bundle under `.dow/aggregations/`; `dow aggregate --list` and `--show <id>`
 retrieve past results.
 
+### Cross-spec suites (the matrix)
+
+`dow aggregate` works within a **single** spec. A robustness sweep, though, often spans
+a whole **matrix** - the same check across several models, domains, or temperatures,
+where each cell is its own spec. `dow suite` aggregates versions drawn from **several**
+specs at once. You declare the participating specs and the project's own aggregators/
+plots in a manifest, `specs/<name>.suite.yaml`:
+
+```yaml
+# specs/robustness_matrix.suite.yaml
+name: robustness_matrix
+specs: [check_llama, check_qwen, check_mistral]   # the specs to draw versions from
+select: all              # all | latest | <tag>
+evaluation:
+  aggregators: [suite_metrics.py:agg_matrix]      # your callables; dow ships none
+  plots: [suite_plots.py:plot_matrix]
+```
+
+Each member keeps its own captured `config`, so your aggregator can bucket by spec /
+model / domain / temperature (the `CohortContext` also carries a parallel `specs` list
+naming each member's spec). Member ids are composite `spec:version`. `select` chooses
+the cohort: `all` (every version of each listed spec - the full matrix), `latest` (each
+spec's newest version), or a tag name (each spec's versions carrying that tag). Runs are
+saved as durable, git-tracked suite bundles (kept separate from single-spec
+aggregations, so they never leak into `dow history`); `dow suite --list` and `--show
+<id>` retrieve them, and `--plot` renders the manifest's plot functions.
+
 ### Pluggable plots
 
 dow can render results to figures without shipping a plotting library: reference your
@@ -195,10 +222,10 @@ evaluation:
     - plots.py:forest_plot          # your matplotlib (or any) code; dow ships none
 ```
 
-Run `dow compare --plot` or `dow aggregate --plot`. dow copies each figure into the
-content-addressed artifact store (`.dow/artifacts/`, git-ignored) and records its hash
-and size; for an aggregation the figure is referenced from the persisted bundle, so it
-stays regenerable while the bytes stay out of git.
+Run `dow compare --plot`, `dow aggregate --plot`, or `dow suite --plot`. dow copies each
+figure into the content-addressed artifact store (`.dow/artifacts/`, git-ignored) and
+records its hash and size; for an aggregation or suite the figure is referenced from the
+persisted bundle, so it stays regenerable while the bytes stay out of git.
 
 ### Non-text outputs
 
@@ -250,10 +277,10 @@ should operate on:
 
 Each tool resolves its project directory from the `project_dir` argument, else
 the `DOW_PROJECT_DIR` environment variable, else the server's current directory.
-The 14 tools mirror the CLI: `dow_list_specs`, `dow_init`, `dow_read_spec`,
+The 15 tools mirror the CLI: `dow_list_specs`, `dow_init`, `dow_read_spec`,
 `dow_write_spec`, `dow_commit`, `dow_compare`, `dow_explain`, `dow_eval`,
-`dow_aggregate`, `dow_history`, `dow_inspect`, `dow_tag`, `dow_tree`, and
-`dow_docs`. They return structured JSON (config diffs, metrics, comparator and
+`dow_aggregate`, `dow_suite`, `dow_history`, `dow_inspect`, `dow_tag`, `dow_tree`,
+and `dow_docs`. They return structured JSON (config diffs, metrics, comparator and
 aggregator results, the version tree, and Mermaid - plus, for text outputs, drift
 scores and verdicts), so a client can run the full edit -> commit -> compare loop.
 

dow/__init__.py

@@ -1 +1 @@
-__version__ = "2.0.4"
+__version__ = "2.1.0"

dow/cli.py

@@ -44,7 +44,9 @@ def _spec_path(name: str) -> Path:
 
 def _spec_files() -> list:
     d = _specs_dir()
-    return sorted(d.glob("*.yaml")) if d.is_dir() else []
+    if not d.is_dir():
+        return []
+    return sorted(f for f in d.glob("*.yaml") if not f.name.endswith(service.SUITE_SUFFIX))
 
 
 def _find_spec_name(name: Optional[str]) -> Optional[str]:
@@ -232,6 +234,39 @@ def aggregate(
     report.print_aggregation(result)
 
 
+@app.command(**_doc("suite"))
+def suite(
+    name: Optional[str] = typer.Argument(
+        None, help="Suite manifest name (specs/<name>.suite.yaml)."),
+    select: Optional[str] = typer.Option(
+        None, "--select", help="Override selection: all | latest | <tag>."),
+    plot: bool = typer.Option(False, "--plot", help="Also run the suite's plot functions and store the figures."),
+    list_: bool = typer.Option(False, "--list", help="List stored suite bundles instead of running one."),
+    show: Optional[str] = typer.Option(None, "--show", help="Print a stored suite bundle by id (e.g. a1)."),
+) -> None:
+    """Aggregate versions across several specs (the check x model x domain x temperature matrix)."""
+    try:
+        sname = service.need_suite(_root(), name)
+    except service.DowError as exc:
+        raise typer.BadParameter(str(exc))
+    if list_:
+        data = service.suite_aggregations(_root(), sname)
+        report.print_aggregation_list(sname, data["aggregations"])
+        return
+    if show:
+        try:
+            bundle = service.get_suite_aggregation(_root(), sname, show)
+        except service.DowError as exc:
+            raise typer.BadParameter(str(exc))
+        report.print_aggregation(bundle)
+        return
+    try:
+        result = service.aggregate_suite(_root(), name=sname, select=select, plot=plot)
+    except service.DowError as exc:
+        raise typer.BadParameter(str(exc))
+    report.print_aggregation(result)
+
+
 @app.command(**_doc("explain"))
 def explain(
     a: Optional[str] = typer.Argument(None, help="First version (default: previous)."),