Skip to content

Is capabilities.aligned_transcript meant to reflect requested vs. actually-delivered alignment? #6493

Description

@onurburak9

Bug Description

We're frequently seeing this warning with default TTS configs on Cartesia and ElevenLabs:

`use_tts_aligned_transcript` is enabled but no agent transcript was returned from tts

Before filing this as a confirmed bug, we want to check whether our reading of the code is correct, since we may be misunderstanding something:

  1. cartesia.TTS(word_timestamps=True) and elevenlabs.TTS(sync_alignment=True) (both defaults) appear to set capabilities.aligned_transcript=True unconditionally at construction, independent of whether the specific model/language combination in use actually returns timing data. Cartesia's own plugin has an internal check for this (tts.py:184-197, warns when the language/model combo doesn't support word_timestamps), but as far as we can tell it doesn't clear the capability flag in that case.
  2. In generation.py, timed_texts_fut looks like it resolves to a real (truthy) Chan object as soon as the TTS node is a streaming AsyncIterable — before any timed-word data has actually arrived:
    if not isinstance(tts_node, AsyncIterable):
        timed_texts_fut.set_result(None)
        return False
    timed_text_ch = aio.Chan[io.TimedString]()
    timed_texts_fut.set_result(timed_text_ch)
  3. In agent_activity.py, this truthiness (not actual data arrival) is what flips read_transcript_from_tts = True for a segment:
    if (
        segment.tts is not None
        and use_aligned_transcript
        and (timed_texts := await segment.tts.timed_texts_fut)
    ):
        transcript = timed_texts
        read_transcript_from_tts = True

So tts.capabilities.aligned_transcript seems to answer "was alignment requested at construction?" rather than "will this specific call return alignment data?" — is that an accurate read of the intent here, or is there a capability-narrowing/fallback mechanism elsewhere that we're missing?

We initially also suspected this could cause the affected segment's text to be dropped from the recorded transcript (via forwarded_text/chat_ctx), but we checked our own persisted session data across several occurrences of this warning and the transcript content was intact every time — so that particular theory doesn't hold up, and we're not raising it here. We mention it only so you know we looked and are not assuming a downstream data-loss impact we haven't actually observed.

Expected Behavior

Unclear to us — hence asking. If the capability flag is meant to reflect "was this requested" rather than "will this be delivered," is the current warning-only behavior (with no other visible effect we could detect) the intended handling? Or should the capability be narrowed to combinations known to work?

Reproduction Steps

  1. Create an AgentSession with tts=cartesia.TTS() (defaults) or tts=elevenlabs.TTS() (defaults).
  2. Run a voice session.
  3. Observe the warning firing on a subset of utterances, with no other apparent effect on transcript content that we could detect.

Operating System: Linux (GKE container)

Models Used: ElevenLabs eleven_multilingual_v2; also reproduces with Cartesia sonic-2 (both default configs)

Package Versions

livekit-agents==1.5.18
livekit-plugins-elevenlabs==<pinned version>
livekit-plugins-cartesia==<pinned version>

Confirmed the same defaults/mechanism are still present on main as of writing.

Session/Room/Call IDs

roomID: sim-55bcf056-c1a9-48ca-a3a6-31176914e54a
jobID: AJ_aSVNf4qQNTSD

Proposed Solution

None — we didn't want to guess at a fix before confirming whether our reading of the mechanism above is even correct.

Additional Context

Related: #3667 (identical warning reported Oct 2025, closed without a root-cause conclusion); #3206/#3423/#3409 (history of use_tts_aligned_transcript auto-enable, revert, and the warning's addition).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions