Skip to content

Commit 94915a1

Browse files
authored
Fix typo for Tau2 eval config (#2210)
### What does this PR do? Type of change: Documentation <!-- Details about the change. --> ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: N/A - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Tau2 evaluation guidance to consistently use “judge” and “user-simulator” terminology. * Clarified rate-limit guidance and the related YAML configuration terminology for telecom evaluations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
1 parent d32c2c2 commit 94915a1

2 files changed

Lines changed: 6 additions & 6 deletions

File tree

plugins/modelopt/skills/evaluation/recipes/env.example

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
5656
# Default 250 (the golden value); lower it only to shorten a debugging run.
5757
# GDPVAL_MAX_TURNS=250
5858

59-
# Tau2 (tau2_bench_telecom) — judger + user-simulator model_ids are hardcoded in
59+
# Tau2 (tau2_bench_telecom) — judge + user-simulator model_ids are hardcoded in
6060
# the recipe; only the shared endpoint URL comes from here
61-
# TAU2_ENDPOINT_URL=https://<your-inference-host>/v1/chat/completions # user + judger
61+
# TAU2_ENDPOINT_URL=https://<your-inference-host>/v1/chat/completions # user + judge
6262

6363
# --- nel-next / harbor agentic benchmarks (Terminal-Bench 2.x, SWE-bench) ---
6464
# Sandboxes run in AWS ECS Fargate (shared NVIDIA harbor infra). AWS creds are

plugins/modelopt/skills/evaluation/recipes/tasks/aa/tau2_bench_telecom.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
## Params
88

99
Tau2 uses the evaluated model as the agent plus a separate user-simulator endpoint;
10-
keep both fixed across runs. The judger (**gpt-oss-120B**) and user-simulator
10+
keep both fixed across runs. The judge (**gpt-oss-120B**) and user-simulator
1111
(**Qwen3 235B**) `model_id`s are hardcoded in the fragment below — swap them for
1212
equivalents on your own endpoint if needed. Only the shared `url`
1313
(`TAU2_ENDPOINT_URL`) comes from `.env` (see `recipes/env.example`) — config, not a
@@ -16,12 +16,12 @@ tau2-bench needs the full `/v1/chat/completions` URL (nemo-skills judges use the
1616
`/v1` base).
1717

1818
For parallelism, we have to throttle to a smaller cap due to the test may be throttled by
19-
user and judger API rate limit. If frequent 429 errors are hit, the reported scores could be much lower.
19+
user and judge API rate limit. If frequent 429 errors are hit, the reported scores could be much lower.
2020

2121
The `parallelism:` field is left as `???` — the right value depends on the
2222
judge and user-simulator endpoints' rate limits, which vary per deployment.
2323
Start with a conservative canary value (e.g. 32–128), watch the logs for 429
24-
errors from the judger/user endpoints, and ramp up if stable. The hard
24+
errors from the judge/user endpoints, and ramp up if stable. The hard
2525
upper bound is 512. After choosing a value, recompute the deployment's
2626
`--max-num-seqs` per the rule in SKILL.md Step 3.
2727

@@ -60,7 +60,7 @@ Use this inside the top-level `evaluation.tasks` list:
6060
model_id: nvidia/qwen/qwen-235b # Qwen3 235B; use an equivalent on your own endpoint if needed
6161
url: <TAU2_ENDPOINT_URL> # from .env (full /v1/chat/completions)
6262
api_key: INFERENCE_API_KEY # env-var name; exported, read by harness
63-
judger:
63+
judge:
6464
model_id: nvidia/openai/gpt-oss-120b # gpt-oss-120B; use an equivalent on your own endpoint if needed
6565
url: <TAU2_ENDPOINT_URL> # from .env (full /v1/chat/completions)
6666
api_key: INFERENCE_API_KEY # env-var name; exported, read by harness

0 commit comments

Comments
 (0)