Skip to content

feat(llm): let an LLM declare chat() stateful - #6626

Open
biztex wants to merge 5 commits into
livekit:mainfrom
biztex:feat/preemptive-generation-safe
Open

feat(llm): let an LLM declare chat() stateful#6626
biztex wants to merge 5 commits into
livekit:mainfrom
biztex:feat/preemptive-generation-safe

Conversation

@biztex

@biztex biztex commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Fixes #5924

Problem

Preemptive generation invokes chat() before the user's turn is committed and discards the result when the final context differs. For a plain LLM that's free — but for the LangGraph adapter, one chat() call runs a full graph turn, tools included. A discarded speculative run still commits whatever side effects the graph performed (checkpoints, external systems), with no rollback — silently corrupting agent state. As the issue puts it, there is no seam to distinguish "generate this speculatively" from "commit this once the turn is confirmed," so any state-mutating graph is unsafe with preemptive generation enabled.

Change

The narrowest seam that makes this safe-by-default (a full speculative/commit protocol would be a much bigger design; this doesn't preclude one later):

  • LLM.stateful property, default False — behavior unchanged for every existing plugin. It describes chat() itself: does the call mutate state that outlives it? That covers an adapter running its own tools and a provider keeping the conversation server-side, both of which make a discarded result unsafe.
  • AgentActivity.on_preemptive_generation skips speculation for a stateful LLM (debug log explains why the latency optimization is off).
  • The LangGraph LLMAdapter declares itself stateful, with a stateful=False constructor flag for graphs that are genuinely side-effect-free or idempotent per turn (documented in the docstring).
  • FallbackAdapter surfaces stateful from its wrapped instances and honours it itself: its background recovery probe drains and discards a full chat(), so it no longer probes stateful instances — they are retried on real traffic instead, and any completed request clears the unavailable flag (announcing only real transitions, which also removes a duplicate available=True event the old probe could emit).

Tests

  • End-to-end in the fake-session harness: with preemptive generation enabled but the LLM declaring itself stateful, turn latency matches the preemptive-disabled path (1.1s vs 0.7s) — speculation demonstrably never ran.
  • Adapter-level: base LLM defaults stateless, LangGraph adapter defaults stateful, opt-out works, FallbackAdapter propagates.
  • Fallback recovery: no probe (and no extra chat()) for a stateful instance, probe still runs for a stateless one, a failed stateful instance is retried on the next request, and a later success restores available with exactly one transition event each way.

stateful replaces the original preemptive_generation_safe per review — it names the property of the LLM rather than the feature consuming it. Kept as a plain property rather than starting an LLMCapabilities struct, since LLM has none today (unlike STT/TTS/VAD/Realtime) and introducing one would touch every plugin constructor; happy to do that separately.

Preemptive generation invokes chat() before the user turn is committed and
discards the result when the final context differs. For a plain LLM that is
free; for the LangGraph adapter a chat() call runs a full graph turn - tools
included - so a discarded speculative run still commits whatever side effects
the graph performed, silently corrupting agent state. There was no seam to
express this, making any state-mutating graph unsafe with preemptive
generation enabled.

Add LLM.preemptive_generation_safe (default True, preserving behavior for
every existing plugin). AgentActivity skips preemptive generation when the
LLM declares itself unsafe, with a debug log. The LangGraph adapter defaults
to unsafe and gains a preemptive_generation_safe constructor flag for graphs
that are genuinely side-effect-free or idempotent per turn.

Fixes livekit#5924
@biztex
biztex requested a review from a team as a code owner July 30, 2026 16:20
devin-ai-integration[bot]

This comment was marked as resolved.

A speculative chat() on the fallback wrapper may run on any wrapped
instance, so the wrapper must declare itself unsafe when any wrapped LLM
does - otherwise a side-effect-heavy model (e.g. a LangGraph graph) placed
behind a failover list would still be called speculatively.
Comment thread livekit-agents/livekit/agents/llm/llm.py Outdated
preemptive_generation_safe named the consumer rather than the property of
the LLM. stateful describes chat() itself - it mutates state that outlives
the call - which is what makes a discarded result unsafe, and covers a
provider keeping the conversation server-side as well as an adapter running
its own tools. Preemptive generation stays the only consumer for now.
devin-ai-integration[bot]

This comment was marked as resolved.

a recovery probe runs a full chat() and discards every chunk, which is the
exact situation the stateful contract exists for: probing a LangGraph
adapter would execute a graph turn and commit its side effects just to test
availability. Stateful instances are no longer probed, and are retried on
real traffic instead so one failure does not drop them (often the primary)
for the rest of the process.
devin-ai-integration[bot]

This comment was marked as resolved.

only the background probe reset it, and stateful instances no longer get a
probe - so one failure left them flagged down forever, logging "all LLMs are
unavailable" on every later turn and never reporting the recovery. A
completed request now marks the instance healthy, and both paths announce
only an actual transition so a probe finishing behind a successful request
does not emit a duplicate event.
@biztex biztex changed the title feat(llm): let an LLM declare chat() unsafe for preemptive generation feat(llm): let an LLM declare chat() stateful Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Plugin] livekit-plugins-langchain doesn't play nicely with pre-emptive generation

2 participants