NVIDIA-NeMo
diff --git a/‎docs/evaluator/agent-eval/gym-runner.mdx‎
Lines changed: 320 additions & 0 deletions b/‎docs/evaluator/agent-eval/gym-runner.mdx‎
Lines changed: 320 additions & 0 deletions
@@ -0,0 +1,320 @@
+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+title: "Evaluate a NeMo Gym Environment"
+description: "Run an existing NeMo Gym environment through NeMo Evaluator's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward."
+---
+
+[NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) is an environment framework for agentic rollouts: a
+**resources-server** provides the environment, an **agent** acts in it, and each rollout carries a
+**reward**. If you already have a Gym environment, the **Gym runner** runs it and scores its reward
+through agent-eval — the same [`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the
+same result and bundle as the [quickstart](/documentation/evaluate-models/agent-eval/quickstart).
+Only the runner changes.
+
+Gym owns execution **and** scoring here. The runner shells out to the `gym` CLI and adapts the
+rollout bundle into trials; `GymRewardMetric` surfaces Gym's per-attempt reward.
+
+<Warning>
+
+Like the Harbor runner, this one is **not** zero-dependency — it shells out to the `gym` CLI:
+
+- **NeMo Gym**, installed into its own `uv` environment (below)
+- **The target environment's own dependencies** — each resources-server ships its own
+  `requirements.txt` (the `mcqa` example needs `tiktoken`)
+- **Model credentials** for the collector, in an `env.yaml` (below)
+
+```bash
+uv venv ~/gym-env --python 3.12
+uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken
+export PATH="$HOME/gym-env/bin:$PATH"
+```
+
+Install Gym into **its own environment** and put that environment's `bin` on `PATH`. Gym imports Ray
+at module load, and `nemo-platform` excludes Ray by constraint, so the two generally cannot share a
+virtualenv. The runner resolves `gym` from `PATH` only — there is deliberately no setting pointing at
+a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a
+platform job, and a local path means nothing on the other side of that boundary. In a job image, the
+image owns `PATH` and this resolves normally.
+
+</Warning>
+
+## Credentials
+
+Gym's collector calls your model endpoint directly. It reads the credentials from an `env.yaml` in
+the directory you run from — **this SDK never reads or handles that file**:
+
+```yaml
+policy_base_url: https://<your-openai-compatible-endpoint>/v1
+policy_api_key: <key>
+policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>
+```
+
+Keep it out of version control. Gym searches the working directory first, then its install root.
+
+## The dataset
+
+A Gym dataset is a **jsonl file**, one row per case. Environments ship their example data inside the
+`nemo-gym` wheel, so the bundled `mcqa` benchmark needs no checkout:
+
+```
+<site-packages>/resources_servers/mcqa/data/example.jsonl
+```
+
+`discover_gym_tasks` turns that file into tasks — one per **distinct** row:
+
+```python
+from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks
+
+tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
+```
+
+Task identity is the row's **content hash**, which has two consequences worth knowing before you
+build a dataset:
+
+- Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data
+  problem.
+- Repeating a row is **not** how you ask for repeated attempts. Use `num_repeats` — attempts are a
+  run-level concern, not a dataset one.
+
+## Run it
+
+```python
+import asyncio
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
+    GymAgentTaskRunner,
+    GymRewardMetric,
+    GymRuntimeConfig,
+    discover_gym_tasks,
+)
+
+tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
+
+runner = GymAgentTaskRunner(
+    config=GymRuntimeConfig(
+        agent="simple_agent",
+        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
+        resources_server="mcqa",
+        num_repeats=2,
+    )
+)
+
+result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
+print(result.summary)
+```
+
+Run it from the directory holding `env.yaml`.
+
+The mapping is:
+
+- one Gym dataset → one run
+- each distinct row → one task
+- each attempt → one trial
+
+So `num_repeats=2` over a 5-row dataset yields 5 tasks and 10 trials.
+
+### Configuration
+
+| Field | Required | Notes |
+|---|---|---|
+| `agent` | yes | agent name to collect rollouts with, e.g. `simple_agent` |
+| `agent_config` | yes | agent config passed to `gym env start`, resolved relative to the Gym install, e.g. `responses_api_agents/simple_agent/configs/simple_agent.yaml` |
+| `resources_server` | yes | resources-server (environment) name, e.g. `mcqa` |
+| `model_type` | no | `inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and fails against chat-only endpoints |
+| `bind_resources_server` | no | auto-bind the agent's `resources_server.name` via a Hydra override |
+| `num_repeats` | no | attempts per row; each attempt becomes one trial |
+| `concurrency` | no | concurrent rollouts during collection — tune to your model endpoint's limits |
+| `hydra_params` | no | parameters merged into Gym's Hydra config, e.g. `{"model": {"temperature": 0.7}}` |
+| `env_vars` | no | environment variables set on the `gym` invocation |
+| `reward_key` | no | key read from each rollout record (default `reward`) |
+| `startup_timeout_s` | no | max wait for `gym env start` readiness |
+| `collection_timeout_s` | no | max wait for collection; `None` is unbounded |
+
+Anything `GymRuntimeConfig` does not expose can go through `hydra_params`, which is flattened to
+Hydra's override grammar and applied to `gym env start`. For the full set of knobs, see the
+[NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym).
+
+## Read the results
+
+Gym's reward arrives as `gym_reward.reward`, and Gym's own aggregates are imported alongside the
+SDK's under a `runner.gym.*` prefix — the prefix is what tells you which side computed a number:
+
+```python
+for score in result.summary.scores.scores:
+    print(score.name)
+# gym_reward.reward
+# gym_reward.reward.pass@1
+# runner.gym.pass@1/accuracy
+# runner.gym.input_tokens
+```
+
+Gym reports accuracy on a **0–100** scale where the SDK uses **0–1**, so `runner.gym.pass@1/accuracy`
+of `50.0` corresponds to a `gym_reward.reward` mean of `0.5`. Trials, scores, and the run bundle are
+otherwise read exactly as in
+[Reading Results](/documentation/evaluate-models/agent-eval/reading-results).
+
+## Output directories
+
+Each run writes to a fresh temporary directory by default. To choose one, set `work_dir` on the run
+config:
+
+```python
+from pathlib import Path
+
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
+
+result = asyncio.run(
+    AgentEvaluator().run(
+        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
+    )
+)
+```
+
+Give every run its own. The runner **refuses** to reuse a directory that already holds Gym rollout
+output, raising `FileExistsError`: Gym appends to its failures sidecar, so reusing one would mix two
+runs together, and the runner raises rather than clearing a previous run's results.
+
+Gym's own artifacts land in a `gym_run/` subdirectory — `rollouts.jsonl`,
+`rollouts_failures.jsonl`, `rollouts_aggregate_metrics.json`, and the materialized
+`gym_input.jsonl` handed to collection.
+
+## How it runs Gym
+
+The runner uses Gym's **two-step** flow, which reads a dataset file directly — no split-driven data
+preparation and no HuggingFace downloads:
+
+1. `gym env start …` brings up the resources-server, agent, and model servers.
+2. `gym eval run --no-serve --input <dataset> …` collects rollouts against them.
+
+The dataset handed to step 2 is **not** your source file. The runner materializes a normalized one
+into the run's work directory, one row per requested task, with `_ng_task_index` stamped explicitly.
+Gym honors a caller-supplied `_ng_task_index` and echoes it back on every rollout record, so rollouts
+join back to tasks through a map the runner owns rather than a guess about Gym's row ordering. That
+is also what lets you run a **subset** of tasks and roll out only that subset.
+
+### Logs
+
+Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for startup,
+and `gym_eval.stdout.log` / `gym_eval.stderr.log` for collection — and mirrored to the
+`nemo_evaluator_sdk.agent_eval.runtimes.gym` logger at `DEBUG`. Startup and collection failures name
+the relevant file and inline its last lines. To watch Gym's output in your own terminal:
+
+```python
+import logging
+
+logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)
+```
+
+## Submit as a platform job
+
+A Gym runner can be submitted as a durable platform job **from the live runner object**, rather than
+described a second time as a job spec — the configuration you validated locally is the configuration
+that runs.
+
+`submit` takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an
+ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that
+`discover_gym_tasks` split across `inputs['gym_row']` and `metadata['gym_row_extras']`. Build the
+tasks with `discover_gym_tasks` and store both halves:
+
+```python
+from nemo_evaluator.api.fields import MetadataItem, MetricInline
+from nemo_evaluator.api.schemas import (
+    EvaluatorTaskDefinition,
+    TaskInput,
+    TaskInputs,
+    TaskRef,
+    TasksetInput,
+    TasksetRef,
+)
+from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
+from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
+    GymAgentTaskRunner,
+    GymRewardMetric,
+    GymRuntimeConfig,
+    discover_gym_tasks,
+)
+from nemo_platform import NeMoPlatform
+
+client = NeMoPlatform(base_url="http://localhost:8080", workspace="default")
+
+runner = GymAgentTaskRunner(
+    config=GymRuntimeConfig(
+        agent="simple_agent",
+        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
+        resources_server="mcqa",
+    )
+)
+
+tasks = discover_gym_tasks("path/to/example.jsonl")
+
+# GymRewardMetric is not a built-in metric type, so it needs the cloudpickle packager.
+reward = MetricInline.model_validate(
+    bundle_metric(GymRewardMetric(), CloudpickleMetricBundlePackager()).model_dump(mode="json")
+)
+
+names = []
+for index, task in enumerate(tasks):
+    name = f"mcqa-{index}"
+    names.append(name)
+    client.evaluator.tasks.create(
+        name,
+        task=TaskInput(
+            spec=EvaluatorTaskDefinition(
+                kind="evaluator",
+                intent=task.intent,
+                inputs=TaskInputs(**task.inputs),
+                metrics=[reward],
+            ),
+            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
+        ),
+    )
+
+client.evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))
+
+job = client.evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
+job.wait_until_done()
+```
+
+Two things that bite here:
+
+- **Do not name the task after `task.id`.** It is a 64-character content hash beginning with a digit,
+  and entity names cap at 63 characters and must start with a letter. Derive a name, as above; the
+  content hash stays the task's own `id`.
+- **`gym_row` rides on `inputs` and `gym_row_extras` on the task's `metadata`** — the field beside
+  `spec`, not inside it. A task missing either is rejected job-side with `task '<id>' is missing
+  inputs['gym_row'] and/or metadata['gym_row_extras']`.
+
+A value with no JSON form — a callable in `hydra_params`, say — is refused with
+`UnsubmittableRunnerError` at submit time, rather than failing inside the transport with an error
+that names neither the runner nor the field.
+
+The returned handle is an `AgentEvaluatorJobResource`. Unlike a dataset-driven job it has no
+`get_result()` or `download_artifacts()`, because an agent evaluation publishes agent-eval results
+and a summary rather than row scores. Read the scores through
+`client.evaluator.agent_eval_results`.
+
+Gym jobs run on their own `nmp-gym-tasks` container image rather than the shared CPU task image,
+because Gym requires Ray. No configuration is needed — the target selects it.
+
+## Caveats
+
+- **Per-environment dependencies are heterogeneous.** `mcqa` needs only `tiktoken`; other Gym
+  environments pull in `torch`, COMET, a GPU, or Docker. Providing a Gym runtime with those installed
+  is the caller's responsibility.
+- **`--no-serve --input` bypasses Gym's data-prep** — prompt templating and dataset materialization.
+  Rows that are already complete, like the bundled `example.jsonl`, are faithful; an environment
+  whose rows need templating would need that step run first.
+- **Service-side execution** — Docker or Kubernetes, Ray provisioning — is out of scope for this SDK
+  path. That is the evaluator plugin's concern.
+
+## Next steps
+
+<Cards>
+  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
+  <Card title="Targets and Runners" href="/documentation/evaluate-models/agent-eval/targets-and-runners" />
+  <Card title="Evaluate a Harbor Task Suite" href="/documentation/evaluate-models/agent-eval/harbor-runner" />
+</Cards>