Add worker_timeout, label and env to eval suite manifests - #5712
Draft
aconchillo wants to merge 3 commits into
Draft
Add worker_timeout, label and env to eval suite manifests#5712aconchillo wants to merge 3 commits into
aconchillo wants to merge 3 commits into
Conversation
PipelineWorker's default idle_timeout_frames only reset on speaking and transcription frames, none of which flow in a text-only conversation, so a healthy text-mode bot (an eval driven with send-text, a chat bot) was cancelled by the 300 s idle timeout mid-conversation. Add LLMFullResponseStartFrame, LLMTextFrame and TTSTextFrame to the default: a pipeline streaming an LLM response is not idle, whatever its modality. PipelineTask shares the default. Claude-Session: https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR
A harness worker was killed after a fixed 600 s whatever the scenario, so a long scripted scenario legitimately running past that lost its whole run. A manifest's `worker_timeout:` (or `-w/--worker-timeout`, the command line winning like the other overrides) now sets the cap, and left unset it is derived from the scenario: a scripted scenario's turn budgets summed (each turn's largest within_ms, or the default timeout) or a simulation's max_duration_s, with the 600 s safety net as the floor and 60 s on top for the judge and teardown. The run carries its scenario's budgets from manifest load, so the cap is computed where the default timeout is known. A suite entry can also carry a `label:` and an `env:`, so one bot file runs under several configurations as separate entries. EvalRun gains `label` (default: the bot path) and `env`; the label is what the dashboard, the failure and pass-rate summaries and results.jsonl (a new "label" field, "bot" kept) show, -p/--pattern matches it as well as the path, and its filesystem-safe slug names the run's artifacts so two configurations of one bot do not overwrite each other's logs. The env is laid over the suite's environment when the bot is spawned. Claude-Session: https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR
Codecov Report❌ Patch coverage is
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Configuration a long benchmark needs from
pipecat eval suite. Running a 30-turn latency benchmark through it hit three limits: the fixed 600 s harness worker cap lost whole runs, the pipeline idle timeout cancelled text-mode conversations after 300 s, and one bot file per model configuration was needed because a manifest entry has no label or environment of its own.worker_timeout:(seconds), overridden by-w/--worker-timeout, capping how long a run's harness worker may take before the suite kills it and reports the run as an error. Left unset, the cap is derived from the scenario — a scripted scenario's turn budgets summed (each turn's largestwithin_ms, or the default timeout), or a simulation'smax_duration_s— with a 600 s floor and 60 s on top for the judge and teardown, so a long scripted scenario is no longer cut off at a fixed 600 s.PipelineWorker's defaultidle_timeout_framescount an LLM response being generated (LLMFullResponseStartFrame,LLMTextFrame,TTSTextFrame) as activity, so a healthy text-only conversation (an eval driven withsend-text, a chat bot) is no longer cancelled by the 300 s idle timeout.PipelineTaskshares the default.label:and anenv:, so one bot file runs under several configurations (a model, a reasoning effort) as separate entries. The label names the entry in the dashboard, the failure and pass-rate summaries,results.jsonl(a new"label"field;"bot"still holds the path) and the artifact filenames, so two configurations of one bot keep separate logs and recordings;-p/--patternmatches it as well as the path.env:is added to the spawned bot's environment over the suite's own.Testing
uv run pytest tests/test_evals_*.py tests/cli/test_eval_display.py tests/cli/test_eval_env.py tests/test_pipeline.pyscripts/release-evals/README.mdunder "One bot, several configurations".https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR