Skip to content

Commit 378adb6

Browse files
nicotnicot
committed
docs: make example agent sandbox setup portable
Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
1 parent 3e6c909 commit 378adb6

1 file changed

Lines changed: 98 additions & 216 deletions

File tree

‎docs/get-started/example-agent.mdx‎

Lines changed: 98 additions & 216 deletions
Original file line numberDiff line numberDiff line change
@@ -3,287 +3,169 @@
33
# SPDX-License-Identifier: Apache-2.0
44

55
title: "Get started with an example agent"
6-
description: "Run the optimization loop from end to end on τ-Bench"
6+
description: "Run the optimization loop from end to end"
77
---
88

9-
This provides a guide for how you can quickly load agent traces into your NeMo Platform, run the analyst agent to discover issues with that agent, and launch an experimentalist to fix those issues. This is meant as an example to show how the whole platform works. You can also jump straight to setting up your agent with NeMo Platform.
9+
This guide runs the NeMo Platform optimization loop on a small example agent
10+
with a known bug. You will run the agent, and record the execution trace to NeMo
11+
Platform, use the Analyst to describe the problem, and run the Experimentalist
12+
to fix the problem.
1013

1114
## Prerequisites
1215

1316
- Git, GNU Make, uv, and pnpm
14-
- Docker Engine 29.6.2 or later
17+
- Docker Engine
1518
- [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) (`sbx`) 0.37.1 or later
16-
- An API key for the inference provider used by the Tau3 agent under test.
19+
- A Docker account signed in through `sbx login`
20+
- An API key for your model provider
1721

18-
## 1. Clone and bootstrap NeMo Platform
22+
## 1. Set up NeMo Platform
1923

20-
This step will walk you through setting up NeMo Platform on your local computer. It will install the platform and its dependencies.
24+
Clone the repository and install its dependencies:
2125

2226
```bash
2327
git clone https://github.com/NVIDIA-NeMo/nemo-platform.git
2428
cd nemo-platform
2529
make bootstrap
2630
```
2731

28-
## 2. Start NeMo Platform
29-
30-
Agent traces are stored in ClickHouse. Make sure Docker Desktop or the Docker daemon is running, then
31-
run setup. Intake automatically provisions and reuses a local ClickHouse container. Setup starts the
32-
platform services in the background and walks you through the configuration required to run the platform:
32+
Make sure Docker is running, then start the Platform:
3333

3434
```bash
3535
uv run nemo setup
3636
```
3737

38-
Register the provider you want the Platform agents to use, then select a
39-
default model for quality-critical work and a fast model for latency-sensitive
40-
work. Press Enter at the fast-model prompt to reuse the default. These
41-
workspace-qualified Model Entities are used by the Analyst, Eval Author, and
42-
Experimentalist; their provider credentials stay in Platform Secrets.
43-
44-
To use an external ClickHouse instead, set `NMP_INTAKE_CLICKHOUSE_URL` before running setup.
45-
46-
You should now be able to navigate to `http://localhost:8080` and see the NeMo Platform web UI.
47-
48-
## 3. Prepare τ-Bench and run the airline agent
49-
50-
Now that we have a running NeMo Platform, it's time to load up some data! This example uses the τ-Bench from Sierra. It's a great representation of a simplified agent that performs a business-critical task. Our first step will be to set up our environment variables so that we can run models properly.
51-
52-
```bash
53-
cp plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.example plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env
54-
```
38+
Follow the prompts to connect a model provider and select the default and fast
39+
models. Press Enter to use the default model for both.
5540

56-
Then, update the `.env` file with the inference endpoint and key for the Tau3
57-
agent under test. If you will run the Experimentalist in a Docker Sandbox,
58-
also uncomment `NEMO_DEFAULT_MODEL` and `NEMO_FAST_MODEL` and copy the
59-
workspace-qualified IDs reported by `nemo setup`. Source the file afterward.
41+
Open [NeMo Studio](http://localhost:8080) when setup finishes.
6042

61-
```bash
62-
source plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env
63-
```
43+
## 2. Record a failing run
6444

65-
Then, we'll run a script to download the benchmark datasets and put it into the appropriate directories:
45+
Create a sandbox and build the smoke agent's small test environment:
6646

6747
```bash
68-
plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-datasets.sh
69-
```
48+
repo="$(git rev-parse --show-toplevel)"
49+
smoke="plugins/nemo-experimentalist/examples/smoke-agent"
50+
workspace="smoke-agent"
51+
platform_url="http://host.docker.internal:8080"
7052

71-
Finally, we'll run our agent using the dataset tasks and record the data to NeMo Platform:
53+
sbx create --clone --name nemo-experimentalist shell "$repo"
54+
sbx policy check network --sandbox nemo-experimentalist localhost:8080 ||
55+
sbx policy allow network --sandbox nemo-experimentalist localhost:8080
56+
sbx exec --user root nemo-experimentalist bash -lc \
57+
'apt-get update && apt-get install -y build-essential python3-dev'
7258

73-
```bash
74-
uv run --frozen plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py
59+
sbx exec --workdir "$repo" \
60+
--env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \
61+
nemo-experimentalist bash -lc "
62+
uv run --no-project $smoke/scripts/build_image.py &&
63+
uv run --frozen --python 3.13 \
64+
--package nemo-experimentalist-plugin \
65+
--with ./plugins/nemo-agents \
66+
python $smoke/scripts/record_traces.py \
67+
--group g1-aggregation \
68+
--split insight-evidence \
69+
--workspace $workspace \
70+
--agent $smoke/agent \
71+
--base-url $platform_url"
7572
```
7673

77-
To run more tasks concurrently, add an explicit value such as `--concurrency 10`.
78-
Each task starts Docker containers, so choose a value appropriate for your
79-
available CPU and memory.
74+
If an organization policy rejects the local allow rule, ask your administrator
75+
to allow `localhost:8080`.
8076

81-
At this point you can navigate to the [traces tab](http://localhost:8080/studio/workspaces/tau3-airline/intake/traces) and see the traces from the agent.
77+
The command runs five tasks that expose the same bug in our agent. Open the
78+
[Traces view](http://localhost:8080/studio/workspaces/smoke-agent/intake/traces)
79+
to see the results.
8280

83-
## 4. Run the Analyst
81+
<Warning>
82+
The Experimentalist runs model-written code. The sandbox protects your working
83+
tree, but it can read the cloned repository and any values passed with `--env`.
84+
</Warning>
8485

85-
Now that we have data in our system, we can run the analyst agent to understand the ways that our agent is performing well, and where it is falling down. The goal of the analyst agent is to crawl production data, and determine where your agent is falling down or disappointing your customers.
86+
## 3. Find the problem
8687

87-
If the analyst discovers issues in your application, it will create what we call an 'insight'. An insight is a human-readable description of a problem in your agentic system. The closest analogy is a bug report. They don't try to describe why an issue happened or the code you should change to fix it, and they should be understandable to a user of your agent.
88+
The Analyst will examine the agent traces and identify significant problems in
89+
your AI application, called an 'Insight'. An insight is a description of a
90+
problem occurring in your AI agent.
8891

89-
Let's run it:
92+
Run the Analyst against the failing traces:
9093

9194
```bash
9295
uv run --frozen nemo agents analyst run \
93-
--agent nemo-experimentalist-tau3-nooa \
94-
--workspace tau3-airline \
95-
--base-url "$NMP_BASE_URL"
96+
--agent smoke-agent \
97+
--agent-spec "$smoke/AGENT-SPEC.md" \
98+
--workspace "$workspace"
9699
```
97100

98-
This will take a few minutes to run. Once it's done, you can navigate to the [insights tab](http://localhost:8080/studio/workspaces/tau3-airline/optimizer) to see the issues the analyst discovered in the τ-Bench Airline Agent.
101+
The Analyst creates an Insight: a short description of the problem. It will also
102+
include the traces where it saw the problem. You can always navigate to the
103+
traces to see the actual execution that the Analyst identified as problematic,
104+
and see if you agree that it is a problem.
99105

100-
## 5. Optimize performance with the experimentalist
101-
102-
The next step is to improve the τ-Bench agent using the experimentalist. The
103-
experimentalist will run your evals, debug failures using trace data, understand
104-
the root cause of the failure and attempt to fix it. After it makes the change,
105-
it will run the evals again to validate whether the change improved the
106-
performance of your agent on the evaluation.
107-
108-
This is a shortened example that only uses a few tasks, but it can still take up
109-
to an hour to finish. First we will set up a new workspace for the optimization process:
106+
Copy the Insight ID shown in square brackets at the end of the command:
110107

111108
```bash
112-
uv run --frozen nemo workspaces create canonical-tau3-airline \
113-
--description "Tau3 Airline Experimentalist runs" \
114-
--exist-ok
109+
insight_id="<Insight ID>"
115110
```
116111

117-
The Experimentalist generates a draft pull request (PR) or merge request (MR)
118-
when it receives a GitHub or GitLab agent source. It creates a branch only when
119-
a changed candidate wins validation.
112+
You can also find it in the
113+
[Insights view](http://localhost:8080/studio/workspaces/smoke-agent/optimizer).
120114

121-
Git integration is optional. To run the optimization without creating a
122-
repository or PR/MR, [jump to the local-only alternative](#optional-run-locally-without-creating-a-pr-or-mr).
115+
## 4. Fix the problem
123116

124-
To optimize the example agent, create a private GitHub repository with the
125-
[GitHub CLI](https://cli.github.com/) and populate it with the example source.
126-
Replace `your-github-user-or-org` with the GitHub account or organization that
127-
will own the repository. `rsync` excludes `.env`, so your API key is not copied
128-
into the repository.
117+
Load the models you selected during setup:
129118

130119
```bash
131-
gh auth login
132-
# Choose HTTPS when prompted for the preferred Git protocol.
133-
export GITHUB_OWNER="your-github-user-or-org"
134-
mkdir -p "$HOME/src"
135-
cd "$HOME/src"
136-
gh repo create "$GITHUB_OWNER/tau3-nooa-agent" --private --clone
137-
export EXAMPLE_AGENT_REPO="$PWD/tau3-nooa-agent"
138-
cd -
139-
rsync -a --exclude='.env' plugins/nemo-experimentalist/examples/tau3-nooa-agent/ "$EXAMPLE_AGENT_REPO/"
140-
git -C "$EXAMPLE_AGENT_REPO" checkout -b main
141-
git -C "$EXAMPLE_AGENT_REPO" add .
142-
git -C "$EXAMPLE_AGENT_REPO" commit -m "Add the tau3 NOOA example agent"
143-
git -C "$EXAMPLE_AGENT_REPO" push -u origin main
144-
export AGENT_REPO_URL="https://github.com/$GITHUB_OWNER/tau3-nooa-agent.git"
120+
read -r NEMO_DEFAULT_MODEL NEMO_FAST_MODEL < <(
121+
uv run --frozen python -c \
122+
'from nemo_platform.config import get_context; c = get_context(); print(c.default_model, c.fast_model or c.default_model)'
123+
)
124+
export NEMO_DEFAULT_MODEL NEMO_FAST_MODEL
145125
```
146126

147-
Run the Experimentalist against the Git repository inside a Docker Sandbox. The
148-
`@main` suffix selects the source ref to optimize. It clones that baseline,
149-
pushes a validated winner to a new branch, and opens a draft PR or MR against
150-
`main`. To use a different target branch, set `storage.pr_base_branch` in the
151-
configuration. Authenticate GitHub inside the sandbox so it can clone and push
152-
the private repository.
153-
154-
<Warning>
155-
Clone mode is an integrity boundary, not a confidentiality boundary. The sandbox
156-
can read the cloned host repository and every credential passed with `sbx exec --env`.
157-
Use dedicated, revocable, spending-limited credentials.
158-
</Warning>
127+
Run the Experimentalist:
159128

160129
```bash
161-
repo="$(git rev-parse --show-toplevel)"
162-
sbx create --clone --name nemo-experimentalist-git shell "$repo"
163-
gh auth token | sbx exec -i --workdir "$repo" nemo-experimentalist-git \
164-
gh auth login --with-token
165-
sbx exec --workdir "$repo" nemo-experimentalist-git gh auth setup-git
166-
sbx exec --workdir "$repo" nemo-experimentalist-git \
167-
git clone "$AGENT_REPO_URL" /tmp/tau3-nooa-agent
168130
sbx exec --workdir "$repo" \
169131
--env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \
170-
--env INFERENCE_API_KEY \
171-
--env INFERENCE_API_BASE \
172-
--env OPENAI_API_KEY \
173-
--env OPENAI_BASE_URL \
174132
--env NEMO_DEFAULT_MODEL \
175133
--env NEMO_FAST_MODEL \
176-
--env TAU2_USER_MODEL \
177-
--env TAU2_NL_ASSERTIONS_MODEL \
178-
--env AUT_MODEL_NAME \
179-
nemo-experimentalist-git \
180-
uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
181-
nemo agents experimentalist run \
182-
--no-insight \
183-
--agent "${AGENT_REPO_URL}@main" \
184-
--agent-spec /tmp/tau3-nooa-agent/AGENT-SPEC.md \
185-
--train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \
186-
--validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \
187-
--workspace canonical-tau3-airline \
188-
--framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
189-
--config /tmp/tau3-nooa-agent/experimentalist-smoke.yaml \
190-
--experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \
191-
--base-url http://host.docker.internal:8080
192-
```
193-
194-
This setup enables `storage.archive_candidates`, which pushes every generated
195-
candidate branch; the default pushes only the winner. After the run, inspect a
196-
non-winning candidate with `git fetch origin`,
197-
`git -C "$EXAMPLE_AGENT_REPO" branch -r --list 'origin/optimizer/*'`, and
198-
`git diff main...origin/optimizer/<run-id>/<candidate>`.
199-
200-
GitLab is supported too: use a GitLab repository URL and authenticate with
201-
`glab auth login`. The Experimentalist opens a draft PR/MR only when it finds a
202-
changed winning candidate.
203-
204-
The trace records include the Experimentalist evaluation ID and Tau3 task ID.
205-
After the run completes, inspect the sandbox's
206-
`plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize/run.json`
207-
for the selected winner and compare the `agent-0` and `agent-1` directories to
208-
review the code change that was evaluated.
209-
210-
### Optional: run locally without creating a PR or MR
211-
212-
Use this alternative if you do not want to create or push to a repository. It
213-
changes only a private sandbox clone and never creates a PR/MR. The sandbox
214-
uses `host.docker.internal` to reach the NeMo Platform services running on the
215-
host:
216-
217-
```bash
218-
repo="$(git rev-parse --show-toplevel)"
219-
sbx create --clone --name nemo-experimentalist shell "$repo"
220-
sbx exec --workdir "$repo" \
221-
--env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \
222-
--env INFERENCE_API_KEY \
223-
--env INFERENCE_API_BASE \
224-
--env OPENAI_API_KEY \
225-
--env OPENAI_BASE_URL \
226-
--env NEMO_DEFAULT_MODEL \
227-
--env NEMO_FAST_MODEL \
228-
--env TAU2_USER_MODEL \
229-
--env TAU2_NL_ASSERTIONS_MODEL \
230-
--env AUT_MODEL_NAME \
231134
nemo-experimentalist \
232-
uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
233-
nemo agents experimentalist run \
234-
--no-insight \
235-
--agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \
236-
--agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \
237-
--train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \
238-
--validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \
239-
--workspace canonical-tau3-airline \
240-
--framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
241-
--config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \
242-
--experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \
243-
--base-url http://host.docker.internal:8080
135+
uv run --frozen --python 3.13 \
136+
--package nemo-experimentalist-plugin \
137+
--with ./plugins/nemo-agents \
138+
nemo agents experimentalist run \
139+
--profile "$smoke/optimizer.yaml" \
140+
--insight "$insight_id" \
141+
--workspace "$workspace" \
142+
--base-url "$platform_url" \
143+
--experiment-dir "$smoke/.nemo-optimizer/example-run" \
144+
--config "$smoke/configs/short.yaml"
145+
146+
mkdir -p tmp
147+
sbx cp "nemo-experimentalist:$repo/$smoke/.nemo-optimizer/example-run" \
148+
tmp/smoke-agent-example-run
244149
```
245-
The sandbox needs outbound access to the package, model, registry, Harbor
246-
dataset, and NeMo Platform endpoints used by the run. `host.docker.internal`
247-
is translated to the host's loopback interface so the sandbox can reach the
248-
NeMo Platform service on port 8080.
249150

250-
<Warning>
251-
Clone mode protects the host checkout from writes, but it does not isolate
252-
secrets or host services from code in the sandbox. The complete host repository,
253-
including ignored `.env` files, is readable at `/run/sandbox/source`. Values
254-
passed with `sbx exec --env` are readable by the optimizer, candidate agents,
255-
Harbor verifier, and their subprocesses. Use dedicated, revocable,
256-
spending-limited credentials, and do not expose unrelated host services or
257-
network destinations. Optimizer and task code can use any outbound access
258-
granted to the sandbox.
259-
</Warning>
151+
The Experimentalist studies the Insight, creates possible fixes, and tests them
152+
against new tasks. Return to the [Insights
153+
view](http://localhost:8080/studio/workspaces/smoke-agent/optimizer) when the
154+
run finishes to compare the original agent with the selected fix. This can take quite a bit of time to run.
260155

261-
The trace records include the Experimentalist evaluation ID and Tau3 task ID.
262-
After the run completes, inspect the run summary and compare the baseline with
263-
the candidate. The `diff` command exits with status 1 when it finds candidate
264-
changes, which is expected:
156+
Once it finishes, you can examine the improved agent in
157+
`tmp/smoke-agent-example-run`.
158+
When you run the experimentalist on a real agent, you would upload these changes
159+
to a Pull Request.
265160

266-
```bash
267-
artifact_dir="plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize"
268-
sbx exec --workdir "$repo" nemo-experimentalist \
269-
sed -n '1,240p' "$artifact_dir/run.json"
270-
sbx exec --workdir "$repo" nemo-experimentalist \
271-
diff -ru "$artifact_dir/agents/agent-0" "$artifact_dir/agents/agent-1"
272-
```
161+
## 5. Clean up
273162

274-
Copy artifacts you want to retain to the host before removing the sandbox:
275-
276-
```bash
277-
mkdir -p tmp/tau3-airline-artifacts
278-
sbx cp "nemo-experimentalist:$repo/$artifact_dir" \
279-
tmp/tau3-airline-artifacts/
280-
```
281-
282-
Stopping preserves the VM, its private Git clone, experiment output, installed
283-
packages, and Docker image/build cache. Remove the sandbox when you no longer
284-
need that state:
163+
Remove the sandbox when you are done:
285164

286165
```bash
287166
sbx stop nemo-experimentalist
288167
sbx rm nemo-experimentalist
289168
```
169+
170+
To run the loop on your own agent, continue with the
171+
[agent onboarding guide](/documentation/get-started).

0 commit comments

Comments
 (0)