add byteplus plugins initial version to livekit - #6647
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| request_transcript_end = await _request_and_emit_audio( | ||
| provider=self._tts, | ||
| session=self._tts._ensure_session(), | ||
| opts=self._opts, | ||
| text=text, | ||
| output_emitter=output_emitter, | ||
| conn_options=self._conn_options, | ||
| request_id=utils.shortuuid(), | ||
| transcript_offset=transcript_offset, | ||
| decode_compressed_audio=decode_compressed_audio, | ||
| ) | ||
| transcript_offset += request_transcript_end | ||
| output_emitter.flush() |
There was a problem hiding this comment.
🟡 Word timings for spoken sentences after the first can be wrong or stacked on top of each other
The running time offset used for word-level captions is advanced by the last word's timestamp instead of the audio actually produced (transcript_offset += request_transcript_end at livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py:579), so captions for later sentences can drift earlier or all land at the same time.
Impact: Live captions drift out of sync with the spoken audio, and if a sentence comes back without timings every following sentence's captions overlap at the same point.
Mechanism: per-request timestamp offset derived from word ends rather than emitted audio duration
In SynthesizeStream._run, each tokenized sentence becomes a separate HTTP request whose provider timestamps restart at 0, so the plugin shifts them with transcript_offset (livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py:568-579). _request_and_emit_audio returns request_transcript_end, computed in _handle_response_event as the maximum word end_time seen for that request (livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py:1178-1185).
Two consequences:
- The audio of a request is generally longer than its last word's end time (trailing silence, and explicitly so when
silence_durationis configured, which appends silence to every request per the constructor docs at lines 231-233). The offset therefore under-counts and every subsequent sentence's word timings are reported earlier than the audio they describe, accumulating with each sentence. - If a request returns no valid word timestamps (e.g. a cached response — see the
use_cachenote that cached responses carry no timestamps),request_transcript_endstays0.0, so the offset does not advance and all following sentences' timestamps are emitted relative to the same base.
A more robust base would be the audio duration actually pushed for the segment (e.g. output_emitter.pushed_duration()), or max(pushed_duration, max_word_end).
Prompt for agents
In livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py, SynthesizeStream._run performs one HTTP request per tokenized sentence and shifts the provider's per-request word timestamps by a running transcript_offset. The offset is advanced by the maximum word end_time returned by that request (_handle_response_event returns transcript_end). This under-counts whenever the synthesized audio is longer than the last word's end time (notably when silence_duration appends trailing silence to each request), causing cumulative drift, and it does not advance at all when a request returns no word timestamps (e.g. cached responses), causing all later sentences' timings to be based on the same offset. Consider deriving the offset from the audio actually emitted for the segment (AudioEmitter.pushed_duration()) or from max(pushed audio duration, max word end) so the offset is monotonic and matches real playback time.
Was this helpful? React with 👍 or 👎 to provide feedback.
| retryable=status in {408, 429} or status >= 500, | ||
| ) | ||
| request_transcript_end = 0.0 | ||
| async for event in _iter_json_events(resp.content): |
There was a problem hiding this comment.
🟡 Streamed speech can be delayed until the whole provider response arrives
The provider's streamed response is consumed with the default line-based reader (_iter_json_events(resp.content) at livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py:695), so audio is only handed over once a newline shows up, and an unusually long chunk aborts the request with an unhandled error.
Impact: If the service does not newline-separate its events, speech is not played until the full response has been received, and very large events can fail the request in a way that is not retried.
Mechanism: `async for` over `aiohttp.StreamReader` iterates lines, not raw chunks
_iter_json_events does async for raw_chunk in content where content is aiohttp.ClientResponse.content (an aiohttp.StreamReader). StreamReader.__aiter__ iterates via readline(), so each yielded item is a newline-terminated line — it waits for a \n (or EOF) before returning data, and raises ValueError when the buffer exceeds the reader limit (here read_bufsize=10 * 1024 * 1024, set at livekit-plugins/livekit-plugins-byteplus/livekit/plugins/byteplus/tts.py:676) without finding a separator.
The function itself is written to be delimiter-agnostic: it keeps an incremental UTF-8 decoder, accumulates a buffer, tolerates chunk boundaries in the middle of JSON, and even handles tuple chunks from iter_chunks() (lines 1055-1090). That intent is defeated by line-buffered iteration: if BytePlus concatenates JSON objects without newlines, nothing is emitted until the response completes, eliminating streaming latency benefits. A ValueError from the reader is also not one of the handled exception types (only asyncio.TimeoutError, APIError, aiohttp.ClientError, OSError are handled at lines 717-731), so it escapes as a non-APIError and bypasses the framework's retry/error reporting path.
Using resp.content.iter_any() (or iter_chunked(...)) yields data as soon as it arrives and removes the line-length limit dependency.
| async for event in _iter_json_events(resp.content): | |
| async for event in _iter_json_events(resp.content.iter_any()): |
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
No description provided.