Measure per-turn latency in scripted evals and record in stereo - #5714
Draft
aconchillo wants to merge 2 commits into
Draft
Measure per-turn latency in scripted evals and record in stereo#5714aconchillo wants to merge 2 commits into
aconchillo wants to merge 2 commits into
Conversation
A latency benchmark on `pipecat.evals` needs the timestamps behind a turn, not only the pass/fail budget `within_ms` puts on them. The harness sees every one it wants, and reports them per turn. - `EvalTurnTiming` on `EvalScriptTurnResult.timing`, written to `results.jsonl` under `turns[].timing` and rebuilt by `_result_from_dict`: milliseconds from the turn's input anchor to `llm_started`, the first LLM token, the end of the LLM response, the first function call, the bot's started/stopped-speaking reports and the harness VAD's onset of its speech, plus the derived `voice_to_voice_ms` and `speech_padding_ms`. The anchor is the send for a text turn and the end of the utterance for a spoken one (`input_duration_ms` is its length); `within_ms` keeps running from the send, and script.py says so. - `EvalTimingObserver` (pipecat.evals.timing), a `BaseObserver` on the eval client's `PipelineWorker`, takes the measures from the harness pipeline's frames, telling the bot's from the user's by the processor that first pushed them: the input transport for the bot's `LLMFullResponseStart/End`, `LLMTextFrame`, `FunctionCallInProgress`, `Bot*Speaking` and `MetricsFrame`; the output transport for the user utterance's `Bot*Speaking` bracket; the user TTS for its `TTSAudioRawFrame`s; the bot-audio aggregator for the `UserStartedSpeakingFrame` that is `bot_speech_onset`. The script driver calls `begin_turn()` once a turn's input is out; the returned record keeps filling in until the next turn. - `EvalClientOutputTransport.write_audio_frame()` returns once the send task has sent the chunk, the way a write to a sound device would, so the base transport's `BotStoppedSpeakingFrame` for the utterance lands when its last chunk went out: that frame is the spoken turn's anchor. A write waits only while a send task exists, so the media sender draining its queue at stop does not hit the write timeout. - The client serializer turns the bot's RTVI `metrics` messages into `MetricsFrame`s (TTFB, processing, LLM usage); the observer keeps them per turn as `timing.bot_metrics`, and the sink stops them like the bot's other frames. - `pipecat eval run -v` prints `ttfb`, and `v2v` for a spoken turn, as one dim line under each turn (a `timing` progress record). - `EvalClientRecorder.write()` writes stereo: the user on the left channel, the bot on the right, each on its own aligned timeline. Claude-Session: https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR
Codecov Report❌ Patch coverage is
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pipecat evalcould only bound a turn's latency withwithin_ms, not measure it, while the harness already has every timestamp such a benchmark wants. Scripted turns now carry atiming(EvalTurnTiming) onEvalScriptTurnResult, written toresults.jsonlunderturns[].timing: milliseconds from the turn's input anchor tollm_started, the first LLM token, the end of the LLM response, the first function call, the bot's started/stopped-speaking reports, and the harness VAD's onset of the bot's speech, plus the derivedvoice_to_voice_msandspeech_padding_ms. A text turn is anchored at the send, a spoken turn at the end of the played utterance (input_duration_msis its length);within_mskeeps its send anchor.metrics(TTFB, processing time, token usage) deserialize intoMetricsFrames in the harness pipeline and are kept per turn astiming.bot_metrics, so the service-measured TTFB sits next to the harness-measured one.EvalTimingObserver(pipecat.evals.timing), attached to the eval client'sPipelineWorker. It tells the bot's frames from the user's by the processor that first pushed them (input transport, output transport, user TTS, bot-audio aggregator), and the script driver callsbegin_turn()once a turn's input is out; the record keeps filling in until the next turn.EvalClientOutputTransport.write_audio_frame()waits until the send task has sent the chunk, the way a write to a sound device would, so the base transport'sBotStoppedSpeakingFramemarks the true end of the user's utterance; that frame is the spoken turn's anchor.pipecat eval run -vprintsttfb, andv2vfor a spoken turn, under each turn.--record-dir) are stereo, the user on the left channel and the bot on the right, for offline turn-taking analysis.Testing
uv run pytest tests/test_evals_*.py tests/cli/test_eval_display.py tests/cli/test_eval_env.py -qtests/test_evals_timing.pydrives the observer with an injected clock; the session integration tests cover a text turn's timing and metrics and a played audio file's re-anchoring; the recorder tests cover the stereo layout.https://claude.ai/code/session_01NgWzoMcT5TQ85tnmFT1BdR