refactor(evaluator)!: collapse the dataset entrypoints into evaluate_dataset (#1202)

* refactor(evaluator)!: collapse the dataset entrypoints into evaluate_dataset

The backend contract carried two dataset entrypoints that differed only in
metric arity and return type: `evaluate` took one metric and returned an
`EvaluationResult`, `evaluate_benchmark` took a list and returned a
`BenchmarkEvaluationResult`. One metric is the degenerate case of several, so
the two collapse into a single `evaluate_dataset(metrics=[...])`.

`Evaluator.run`/`run_sync` lose their single-metric overloads — three each —
leaving the three that discriminate on config and target, where the choice
actually matters. Callers pass a list.

`BenchmarkEvaluationResult` gains the `__str__` that `EvaluationResult` already
had. Without it, collapsing to one result type would have quietly regressed
`print(result)` to a pydantic dump for every caller.

Two smaller things fall out of having one destination rather than two. The
plugin's evaluate job no longer unwraps a one-metric list to hit the singular
overload. And a backend's flavour is now decided by a single method, so the
async/sync/mixed three-way collapses to two: a lone method is either a
coroutine function or it is not, and there is no mixed client left to reject.

This frees the name `evaluate` for the taskset entrypoint, which follows
separately; nothing claims it here.

BREAKING CHANGE: `EvaluationBackend.evaluate` and `evaluate_benchmark` are
replaced by `evaluate_dataset`, which takes a metric list and returns a
`BenchmarkEvaluationResult`. `Evaluator.run` and `run_sync` no longer accept a
bare metric.

Signed-off-by: Sandy Chapman <schapman@nvidia.com>

* fix(evaluator): repair the callers and exports the dataset collapse left behind

The collapse removed the single-metric overloads but left seven `run_sync`
calls in the high-level walkthrough passing a bare metric. They fail as
`AttributeError: 'tuple' object has no attribute 'output_spec'`, which says
nothing about the real problem. The earlier sweep matched `metrics=[` and these
pass named variables, so none of them matched.

`BenchmarkEvaluationResult` is now what every dataset run returns, but it was
only reachable from `nemo_evaluator_sdk.values.multi_metric_results` — callers
could not name the type they are handed. It is exported from the package root
and from `values` alongside `EvaluationResult`.

The walkthrough's directory example passed `pattern=`, which no `Evaluator`
method has ever accepted, on this branch or before it. The loader already
handles a glob path when no pattern is given, so the example points `dataset`
at `part-*.jsonl` and keeps excluding the file it is meant to exclude.

The skill reference still promised a bare metric and one of two result types.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

* Attach NVSkills validation signatures

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

---------

Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

Author: 3 people

packages/nemo_evaluator_sdk/examples/examples.py

@@ -360,7 +360,7 @@ async def run_offline_local_exact_match_example() -> None:
     print("Running offline exact match...")
 
     exact_match_result = await evaluator.run(
-        metrics=exact_match,
+        metrics=[exact_match],
         dataset=OFFLINE_EXACT_MATCH_DATASET,
         config=RunConfig(parallelism=4),
     )
@@ -382,7 +382,7 @@ async def run_online_local_exact_match_example() -> None:
     print("Running local online exact match...")
 
     exact_match_result = await evaluator.run(
-        metrics=exact_match,
+        metrics=[exact_match],
         target=model,
         dataset=ONLINE_EXACT_MATCH_DATASET,
         prompt_template=ONLINE_CHAT_PROMPT_TEMPLATE,
@@ -523,7 +523,7 @@ async def run_local_metric_with_template_failure_example() -> None:
     print("\nRunning local metric evaluation with an invalid metric template...")
     try:
         await evaluator.run(
-            metrics=invalid_metric,
+            metrics=[invalid_metric],
             dataset=dataset,
             config=RunConfig(parallelism=1),
         )
@@ -553,7 +553,7 @@ async def run_offline_local_llm_judge_example() -> None:
     print("\nRunning local LLM judge evaluation...")
 
     llm_judge_result = await evaluator.run(
-        metrics=llm_judge_metric,
+        metrics=[llm_judge_metric],
         dataset=OFFLINE_JUDGE_DATASET,
         config=RunConfig(parallelism=2),
     )
@@ -575,7 +575,7 @@ async def run_online_local_llm_judge_example() -> None:
     print("\nRunning local online LLM judge evaluation...")
 
     llm_judge_result = await evaluator.run(
-        metrics=llm_judge_metric,
+        metrics=[llm_judge_metric],
         target=model_with_custom_headers,
         dataset=ONLINE_JUDGE_DATASET,
         prompt_template=ONLINE_CHAT_PROMPT_TEMPLATE,
@@ -593,7 +593,7 @@ def run_sync_example() -> None:
 
     evaluator = Evaluator()
     result = evaluator.run_sync(
-        metrics=ExactMatchMetric(reference="{{item.reference}}", candidate="{{item.actual}}"),
+        metrics=[ExactMatchMetric(reference="{{item.reference}}", candidate="{{item.actual}}")],
         dataset=OFFLINE_EXACT_MATCH_DATASET[:1],  # Only run the first sample
         config=RunConfig(parallelism=1),
     )

packages/nemo_evaluator_sdk/examples/high_level_evaluate_walkthrough.ipynb

@@ -40,7 +40,7 @@
     "from IPython.display import Markdown, display\n",
     "from nemo_evaluator_sdk.execution.evaluator import Evaluator\n",
     "from nemo_evaluator_sdk.metrics.exact_match import ExactMatchMetric\n",
-    "from nemo_evaluator_sdk.values import DatasetRows, EvaluationResult"
+    "from nemo_evaluator_sdk.values import BenchmarkEvaluationResult, DatasetRows"
    ]
   },
   {
@@ -55,7 +55,7 @@
     "- [Input mode 2: DatasetRows](#datasetrows)\n",
     "- [Input mode 3: pyarrow.Table](#pyarrow-table)\n",
     "- [Input mode 4: single file path](#file-path)\n",
-    "- [Input mode 5: directory + pattern](#directory-pattern)\n",
+    "- [Input mode 5: directory + glob](#directory-pattern)\n",
     "- [Result APIs showcase](#result-apis)\n"
    ]
   },
@@ -91,7 +91,7 @@
     "    display(pd.DataFrame.from_records(records))\n",
     "\n",
     "\n",
-    "def summarize_run(name: str, result: EvaluationResult) -> dict[str, object]:\n",
+    "def summarize_run(name: str, result: BenchmarkEvaluationResult) -> dict[str, object]:\n",
     "    aggregate = result.to_records(view=\"aggregate\")\n",
     "    exact = next((row for row in aggregate if row.get(\"name\") == \"exact_match\"), aggregate[0] if aggregate else {})\n",
     "    return {\n",
@@ -122,13 +122,13 @@
     "\n",
     "evaluator = Evaluator()\n",
     "\n",
-    "runs: dict[str, EvaluationResult] = {}\n",
+    "runs: dict[str, BenchmarkEvaluationResult] = {}\n",
     "\n",
     "workspace_snapshot = [\n",
     "    {\"path\": str(file_path), \"purpose\": \"Single-file input\"},\n",
     "    {\"path\": str(dataset_dir / \"part-000.jsonl\"), \"purpose\": \"Sharded input part 1\"},\n",
     "    {\"path\": str(dataset_dir / \"part-001.jsonl\"), \"purpose\": \"Sharded input part 2\"},\n",
-    "    {\"path\": str(dataset_dir / \"ignored.jsonl\"), \"purpose\": \"Intentionally excluded question/answer pattern\"},\n",
+    "    {\"path\": str(dataset_dir / \"ignored.jsonl\"), \"purpose\": \"Intentionally excluded from the glob\"},\n",
     "]\n",
     "\n",
     "display(Markdown(f\"**Workspace:** `{workspace}`\"))\n",
@@ -169,7 +169,7 @@
     "# ExactMatchMetric\n",
     "exact_metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ prediction }}\")\n",
     "exact_result = evaluator.run_sync(\n",
-    "    metrics=exact_metric,\n",
+    "    metrics=[exact_metric],\n",
     "    dataset=[\n",
     "        {\"expected\": \"Paris\", \"prediction\": \"Paris\"},\n",
     "        {\"expected\": \"Jupiter\", \"prediction\": \"Saturn\"},\n",
@@ -191,7 +191,7 @@
     "# F1Metric\n",
     "f1_metric = F1Metric(reference=\"{{ reference }}\", candidate=\"{{ prediction }}\")\n",
     "f1_result = evaluator.run_sync(\n",
-    "    metrics=f1_metric,\n",
+    "    metrics=[f1_metric],\n",
     "    dataset=[\n",
     "        {\"reference\": \"The Eiffel Tower is in Paris.\", \"prediction\": \"Eiffel Tower is in Paris\"},\n",
     "        {\"reference\": \"Python is a programming language.\", \"prediction\": \"Python is a snake\"},\n",
@@ -210,7 +210,7 @@
     "# BLEUMetric\n",
     "bleu_metric = BLEUMetric(references=[\"{{ reference }}\"], candidate=\"{{ prediction }}\")\n",
     "bleu_result = evaluator.run_sync(\n",
-    "    metrics=bleu_metric,\n",
+    "    metrics=[bleu_metric],\n",
     "    dataset=[\n",
     "        {\"reference\": \"the cat is on the mat\", \"prediction\": \"the cat is on the mat\"},\n",
     "        {\"reference\": \"a dog runs in the park\", \"prediction\": \"dog runs at park\"},\n",
@@ -234,7 +234,7 @@
     "    epsilon=0.01,\n",
     ")\n",
     "number_result = evaluator.run_sync(\n",
-    "    metrics=number_metric,\n",
+    "    metrics=[number_metric],\n",
     "    dataset=[\n",
     "        {\"expected_value\": \"3.1416\", \"model_value\": \"3.1410\"},\n",
     "        {\"expected_value\": \"2.50\", \"model_value\": \"2.61\"},\n",
@@ -253,7 +253,7 @@
     "# ROUGEMetric\n",
     "rouge_metric = ROUGEMetric(reference=\"{{ reference_summary }}\", candidate=\"{{ model_summary }}\")\n",
     "rouge_result = evaluator.run_sync(\n",
-    "    metrics=rouge_metric,\n",
+    "    metrics=[rouge_metric],\n",
     "    dataset=[\n",
     "        {\n",
     "            \"reference_summary\": \"The launch succeeded after two delays caused by weather.\",\n",
@@ -282,7 +282,7 @@
     "    right_template=\"{{ required_phrase }}\",\n",
     ")\n",
     "string_result = evaluator.run_sync(\n",
-    "    metrics=string_metric,\n",
+    "    metrics=[string_metric],\n",
     "    dataset=[\n",
     "        {\"answer_text\": \"The SLA is 99.9% uptime.\", \"required_phrase\": \"99.9%\"},\n",
     "        {\"answer_text\": \"Support is available weekdays only.\", \"required_phrase\": \"24/7\"},\n",
@@ -307,7 +307,7 @@
     "\n",
     "tool_metric = ToolCallingMetric(reference=\"{{item.reference}}\")\n",
     "tool_result = evaluator.run_sync(\n",
-    "    metrics=tool_metric,\n",
+    "    metrics=[tool_metric],\n",
     "    dataset=[\n",
     "        {\n",
     "            \"reference\": [{\"function\": {\"name\": \"sum\", \"arguments\": {\"x\": 1, \"y\": 2}}}],\n",
@@ -349,7 +349,7 @@
     "]\n",
     "\n",
     "metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ model_output }}\")\n",
-    "inline_result = evaluator.run_sync(metrics=metric, dataset=inline_rows)\n",
+    "inline_result = evaluator.run_sync(metrics=[metric], dataset=inline_rows)\n",
     "\n",
     "runs[\"Inline list[dict]\"] = inline_result\n",
     "\n",
@@ -383,7 +383,7 @@
     ")\n",
     "\n",
     "metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ model_output }}\")\n",
-    "dataset_rows_result = evaluator.run_sync(metrics=metric, dataset=dataset_rows)\n",
+    "dataset_rows_result = evaluator.run_sync(metrics=[metric], dataset=dataset_rows)\n",
     "\n",
     "runs[\"DatasetRows\"] = dataset_rows_result\n",
     "\n",
@@ -410,7 +410,7 @@
     "arrow_table = pa.Table.from_pylist(base_rows)\n",
     "\n",
     "metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ prediction }}\")\n",
-    "arrow_result = evaluator.run_sync(metrics=metric, dataset=arrow_table)\n",
+    "arrow_result = evaluator.run_sync(metrics=[metric], dataset=arrow_table)\n",
     "\n",
     "runs[\"pyarrow.Table\"] = arrow_result\n",
     "\n",
@@ -435,7 +435,7 @@
    "outputs": [],
    "source": [
     "metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ prediction }}\")\n",
-    "file_result = evaluator.run_sync(metrics=metric, dataset=file_path)\n",
+    "file_result = evaluator.run_sync(metrics=[metric], dataset=file_path)\n",
     "\n",
     "runs[\"File path\"] = file_result\n",
     "\n",
@@ -448,9 +448,9 @@
    "source": [
     "<a id=\"directory-pattern\"></a>\n",
     "\n",
-    "## Input Mode 5: Directory Path + `pattern`\n",
+    "## Input Mode 5: Directory Path + Glob\n",
     "\n",
-    "For sharded datasets, pass the directory and target only matching files.\n"
+    "For sharded datasets, point `dataset` at a glob path to target only matching files.\n"
    ]
   },
   {
@@ -460,9 +460,9 @@
    "outputs": [],
    "source": [
     "metric = ExactMatchMetric(reference=\"{{ expected }}\", candidate=\"{{ prediction }}\")\n",
-    "directory_result = evaluator.run_sync(metrics=metric, dataset=dataset_dir, pattern=\"part-*.jsonl\")\n",
+    "directory_result = evaluator.run_sync(metrics=[metric], dataset=dataset_dir / \"part-*.jsonl\")\n",
     "\n",
-    "runs[\"Directory + pattern\"] = directory_result\n",
+    "runs[\"Directory + glob\"] = directory_result\n",
     "\n",
     "directory_result.print_summary(max_rows=2)"
    ]
@@ -475,7 +475,7 @@
     "\n",
     "## Result APIs\n",
     "\n",
-    "All runs return `EvaluationResult` with the same helper surface.\n"
+    "All runs return `BenchmarkEvaluationResult` with the same helper surface.\n"
    ]
   },
   {
@@ -519,7 +519,7 @@
     "\n",
     "- `evaluator.run_sync(metrics=..., dataset=...)` gives one clean API across inline rows, Arrow data, and local files.\n",
     "- Templates (`reference`, `candidate`) handle the mapping between dataset columns and metric inputs.\n",
-    "- `EvaluationResult` supports both human-friendly summaries and dataframe/table pipelines.\n",
+    "- `BenchmarkEvaluationResult` supports both human-friendly summaries and dataframe/table pipelines.\n",
     "- The exact same workflow scales from local notebook demos to production data slices."
    ]
   }

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/__init__.py

@@ -73,6 +73,7 @@
         BooleanValue,
         CandidateOutput,
         ContinuousScore,
+        BenchmarkEvaluationResult,
         DatasetRow,
         DatasetRows,
         DiscreteScore,
@@ -166,6 +167,7 @@ def _resolve_version() -> str:
     "BooleanValue": ".values",
     "CandidateOutput": ".values",
     "ContinuousScore": ".values",
+    "BenchmarkEvaluationResult": ".values",
     "DatasetRow": ".values",
     "DatasetRows": ".values",
     "DiscreteScore": ".values",
@@ -206,6 +208,7 @@ def _resolve_version() -> str:
     "RunConfig",
     "RunConfigOnline",
     "RunConfigOnlineModel",
+    "BenchmarkEvaluationResult",
     "EvaluationResult",
     "Evaluator",
     "ExactMatchMetric",

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/README.md

@@ -22,7 +22,7 @@ The execution package exposes a single public entrypoint:
 # Local SDK execution
 evaluator = Evaluator()
 result = await evaluator.run(
-    metrics=ExactMatchMetric(reference="{{item.reference}}"),
+    metrics=[ExactMatchMetric(reference="{{item.reference}}")],
     dataset=[{"reference": "Paris", "output_text": "Paris"}],
 )
 ```

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/backends/base.py

@@ -21,44 +21,13 @@
     RunConfigOnlineModel,
 )
 from nemo_evaluator_sdk.values.multi_metric_results import BenchmarkEvaluationResult
-from nemo_evaluator_sdk.values.results import AggregateFieldName, EvaluationResult
+from nemo_evaluator_sdk.values.results import AggregateFieldName
 
 BackendParams = RunConfig | RunConfigOnline | RunConfigOnlineModel
 
 
 class EvaluationBackend(Protocol):
-    async def evaluate(
-        self,
-        *,
-        metric: Metric,
-        dataset: DatasetInput | str | Path,
-        params: BackendParams,
-        target: Model | Agent | None = None,
-        field_mapping: FieldMapping | None = None,
-        prompt_template: str | dict[str, Any] | None = None,
-        aggregate_fields: tuple[AggregateFieldName, ...] | None = None,
-        preprocess_hooks: tuple[PreprocessRequest, ...] | None = None,
-        postprocess_hooks: tuple[PostprocessResponse, ...] | None = None,
-    ) -> EvaluationResult:
-        """Evaluate one metric directly and return the completed result.
-
-        Args:
-            metric: Metric to prepare and execute.
-            dataset: Inline dataset rows, a dataset file, or a dataset directory/glob path.
-            params: Validated run configuration for the selected target mode.
-            target: Optional model or agent used to generate candidate responses before scoring.
-            field_mapping: Optional mapping from canonical evaluator fields to dataset columns.
-            prompt_template: Optional prompt template for online target generation.
-            aggregate_fields: Optional aggregate score fields to keep in the returned result.
-            preprocess_hooks: Optional request preprocess hooks for online execution.
-            postprocess_hooks: Optional response postprocess hooks for online execution.
-
-        Returns:
-            The completed single-metric evaluation result.
-        """
-        ...
-
-    async def evaluate_benchmark(
+    async def evaluate_dataset(
         self,
         *,
         metrics: Sequence[Metric],
@@ -91,38 +60,7 @@ async def evaluate_benchmark(
 
 
 class SyncEvaluationBackend(Protocol):
-    def evaluate(
-        self,
-        *,
-        metric: Metric,
-        dataset: DatasetInput | str | Path,
-        params: BackendParams,
-        target: Model | Agent | None = None,
-        field_mapping: FieldMapping | None = None,
-        prompt_template: str | dict[str, Any] | None = None,
-        aggregate_fields: tuple[AggregateFieldName, ...] | None = None,
-        preprocess_hooks: tuple[PreprocessRequest, ...] | None = None,
-        postprocess_hooks: tuple[PostprocessResponse, ...] | None = None,
-    ) -> EvaluationResult:
-        """Evaluate one metric directly and return the completed result.
-
-        Args:
-            metric: Metric to prepare and execute.
-            dataset: Inline dataset rows, a dataset file, or a dataset directory/glob path.
-            params: Validated run configuration for the selected target mode.
-            target: Optional model or agent used to generate candidate responses before scoring.
-            field_mapping: Optional mapping from canonical evaluator fields to dataset columns.
-            prompt_template: Optional prompt template for online target generation.
-            aggregate_fields: Optional aggregate score fields to keep in the returned result.
-            preprocess_hooks: Optional request preprocess hooks for online execution.
-            postprocess_hooks: Optional response postprocess hooks for online execution.
-
-        Returns:
-            The completed single-metric evaluation result.
-        """
-        ...
-
-    def evaluate_benchmark(
+    def evaluate_dataset(
         self,
         *,
         metrics: Sequence[Metric],