Skip to content

Commit e39e44f

Browse files
SandyChapmanclaude
andcommitted
refactor(evaluator): narrow the agent-eval run seam without an assert
The assert target is not None in AgentEvaluator.run was dead at runtime -- the both/neither cases were already rejected above -- and existed only so the type checker could narrow target. Reject the "both" case up front and let the branch chain itself narrow each arm, with the final arm covering "neither". Adds the missing regression test for the neither-supplied case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
1 parent ce2e986 commit e39e44f

3 files changed

Lines changed: 21 additions & 12 deletions

File tree

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -166,17 +166,19 @@ async def run(
166166
runtime_config = resolved_config.model_copy(update={"run_id": run_id})
167167
started_at = datetime.now(UTC)
168168

169-
if (trials is None) == (target is None):
170-
raise ValueError("provide exactly one of trials or target")
169+
seam_error = "provide exactly one of trials or target"
170+
if trials is not None and target is not None:
171+
raise ValueError(seam_error)
171172

172173
async with begin_evaluation_session():
173-
# Branch on which seam was supplied so the type checker can narrow ``target`` to a
174-
# concrete ``AgentEvalTarget`` without a cast.
174+
# Branch on which seam was supplied so the type checker narrows each of ``trials`` and
175+
# ``target`` to a concrete type without a cast. The final arm is the "neither" case.
175176
if trials is not None:
176177
trial_list = list(trials)
177-
else:
178-
assert target is not None
178+
elif target is not None:
179179
trial_list = await self._generate_trials(tasks=task_list, target=target, config=runtime_config)
180+
else:
181+
raise ValueError(seam_error)
180182
scores = await self._score_trials(
181183
tasks=task_list,
182184
trials=trial_list,

packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,11 @@ def test_run_rejects_trials_and_target_together() -> None:
296296
)
297297

298298

299+
def test_run_rejects_neither_trials_nor_target() -> None:
300+
with pytest.raises(ValueError, match="provide exactly one"):
301+
AgentEvaluator().run_sync(tasks=[_task()])
302+
303+
299304
@pytest.mark.asyncio
300305
async def test_run_writes_nothing_until_persist_is_called(tmp_path: Path) -> None:
301306
# The point of the change: computing an evaluation and storing one are separate decisions, so a

sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py

Lines changed: 8 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)