Pair force-completed text with a progress frame, drop unmatched buffered words

[dakshdua]

Changes

force_complete leaves a turn's word tracking in two inconsistent states, both of which surface as the progress view and the TTSTextFrames disagreeing about what the turn said.

A force-completed slot's remainder goes out with no progress frame

The remaining text is emitted as a word frame, but nothing moves the cursors, so the progress view stops where the provider stopped reporting. On a turn whose word stream is cut partway, the word frames cover the full text while the progress transcript ends mid-sentence.

WordCompletionTracker.take_remaining_as_spoken() moves the user-facing and LLM cursors to the end — the same thing the tracker already does when a word completes a slot — and a progress frame goes out alongside the word frame.

A word that matches no slot stays buffered forever

In streaming mode a word event that no slot matches is buffered, waiting for a slot to be promoted to match it against (aggregated_frame_sequencer.py:522-526). _drain_buffered_words runs only on promotion, so at the end of a turn the event stayed buffered and was discarded at the next reset, with nothing logged. Driving 您好。 as three tokens through a streaming sequencer:

after '好'      : frames=['您','好']   buffered=[]
after '。'      : frames=['您','好']   buffered=['。']
force_complete : []
buffered at end: ['。']               warnings: none

A context that ends now reports what it leaves behind — a warning naming the word, the way sentence mode already reports one it cannot place — and drops it. No frame is emitted: a word no slot matched has no place in the turn's text. Another context's buffered words are left alone for their own turn.

Both are gated on self._streaming, since sentence mode never buffers a word in the first place.

Testing

tests/test_aggregated_frame_sequencer.py adds TestForceCompleteWordStream: a buffered word is reported and dropped when its context ends, another context's buffered word is left alone, and a forced tail reports progress to the end.

599 tests pass across the word-timestamp and text-tracking files (test_aggregated_frame_sequencer, test_word_completion_tracker, test_text_segment_map, test_cartesia_tts, the ElevenLabs/Inworld/Soniox TTS files, the context aggregators and test_tts_zero_audio_contexts).

End to end, driving CartesiaTTSService against the live Cartesia websocket with sonic-3.6, comparing main against this branch over seven Chinese and Japanese scenarios on both the streamed-LLM and TTSSpeakFrame routes, the progress-frame half fixes four cases and regresses none:

• a Japanese turn cut after 9 word-timestamp entries has its progress transcript reach the end of the text, matching the word frames, where before it stopped at '確認いたします…ゼ'
• the same holds for a Japanese turn with a spaced em dash cut after 5 entries
• two Japanese turns carrying inline <speed>, <emotion>, <break> and <spell> markup now reproduce the whole turn, markup included, in the progress transcript

Reviewing

uv run pytest tests/test_aggregated_frame_sequencer.py. Either half is reachable from any provider that reports punctuation as its own word-timestamp entry, or that stops reporting entries mid-turn.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
...ipecat/utils/context/aggregated_frame_sequencer.py	98.86% <100.00%> (+0.05%)	⬆️
...c/pipecat/utils/context/word_completion_tracker.py	100.00% <100.00%> (ø)

... and 64 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.