evals: judge function calls, and show the judge the calls the bot made - #5713
Draft
aconchillo wants to merge 3 commits into
Draft
evals: judge function calls, and show the judge the calls the bot made#5713aconchillo wants to merge 3 commits into
aconchillo wants to merge 3 commits into
Conversation
`eval:` is accepted on `function_call` and `function_call_stopped` expectations alongside `calls:`. Each call matched by name (and by any verbatim `args:`) is put to the judge with a call-specific ask that names the function and gives its arguments as compact JSON, over the conversation so far. A "no" fails the expectation with kind `judge_no` and the judge's reason; a "continue" counts as a "no", since a call is not a partial reply. The parser accepts `eval:` on these two events without a warning, and `required_report_level()` asks for `full` whenever a judged scenario asserts on calls, so the judge sees the arguments. Every `function_call` the matcher pops from the stream, claimed or buffered as pending, is added once to the judge's conversation as an assistant message `[tool call] name(args)`, in arrival order with the reply's segments, and the judge's instruction says what such a line is. A reply's `eval:` can therefore check that a confirmation matches what the bot actually submitted. `EvalJudge` gains `add_tool_call()` and `evaluate_call()`, and `format_tool_call()` formats a call as one line. Claude-Session: https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
eval:is accepted onfunction_callandfunction_call_stoppedexpectations alongsidecalls:. Each call the expectation matches by name (and by any verbatimargs:) is put to the judge by name and arguments, over the conversation so far, so a scenario can check whatargs:cannot match word for word — "the suggestion is for a session about OpenTelemetry tracing, submitted for Jennifer Smith". A rejected call fails the expectation with kindjudge_noand the judge's reason; acontinuecounts as ano, since a call is not a partial reply.function_callthe matcher pops goes into the judge's conversation as an assistant message[tool call] name(args), in arrival order with the reply's segments, so aresponsecriterion can check that a confirmation matches what the bot actually submitted.fullfunction-call report level, so the judge sees the arguments either way.EvalJudgegainsadd_tool_call()andevaluate_call();pipecat.evals.judge.format_tool_call()formats a call as one line.Testing
uv run pytest tests/test_evals_*.py -qhttps://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR