Add OpenAILiveLLMService (OpenAI Live API, gpt-live-1) with both delegation modes - #5688
Merged
Conversation
OpenAI's Live API (gpt-live-1) is a full-duplex speech-to-speech model: it listens and speaks at the same time, handles being interrupted on its own, and delegates search, reasoning and tool use to a backend text model while the conversation continues. This adds a WebSocket service for it alongside the Realtime service, plus the Responses delegation mode where OpenAI hosts the backend model. Because turn taking is model-internal, the service never broadcasts interruptions: it proposes user turns from the API's projected transcript turns and recommends ExternalUserTurnStrategies with interruptions disabled, so the aggregator pair records both sides of the conversation without cutting anything off. Every tool therefore runs to completion regardless of cancel_on_interruption; cancellable_by_llm remains the way to let the model stop a tool on request. Function calls made by the delegated Responses model are executed with the handlers registered for the pipeline context's tools, and their outputs are sent to the API as the result frames pass through the service rather than via the context round-trip the other realtime services use. With a full- duplex model a result usually lands while the bot is speaking filler or the user is talking, which is exactly when the assistant aggregator defers (or, in realtime mode, skips) the context push that round-trip depends on. The delegation configuration nests the same OpenAIResponsesLLMSettings the Responses service takes; tools and tool_choice come from the LLMContext. Session configuration is derived from the context on the first LLMContextFrame: a leading system message becomes the instructions and the remaining text messages seed the session as initial_items.
…on() The Live API only takes conversation history at session start, so restoring a saved conversation into a running session means closing it and opening a new one seeded from the context as it is now. reset_conversation() does that, mirroring the Realtime service's method of the same name. The persistent-context example saves the context the aggregators recorded from the session's transcripts and the backend's tool calls, and loads it back through tools the delegated Responses model calls.
With client delegation the live model hands a natural-language request to the application and expects the result back as context. This adds OpenAILiveLLMService.ClientDelegation, backed by a new BackendLLMWorker: a worker that runs any Pipecat LLM service with its own context and aggregator pair, so multi-step tool calling works as it does in any pipeline. The service registers the worker as a child of the pipeline worker and hands each delegation off over the job API through run_backend_job(); every response the backend produces along the way is appended to the delegation as speakable context. The live model doesn't share its conversation with the backend, so the service ships the finished transcript turns since the previous delegation with each request. The worker renders them as a labelled transcript inside the task's user message rather than as messages of their own, the shape OpenAI's reference clients use: the backend's context then holds only what the backend itself said as assistant messages. A run is finished when every LLM run it triggered has ended and no tool call is in flight. Runs are counted as the LLM picks up context frames (with a check of its queue for one still waiting), which stays exact when a tool returns before the response that called it has ended. The job is sequential, matching the reference clients' serialization of backend runs that share one conversation. The job contract is generic — a task plus the turns the backend hasn't seen, text updates back — so a cascade bot can delegate to the same worker from a tool.
The same backend worker and job contract OpenAILiveLLMService uses for client delegation work for any frontend. This example keeps the conversation on a fast cascade pipeline (STT, a small model, TTS) with a single delegate tool that runs the backend job, streams the backend's intermediate responses into the conversation as developer messages, and returns the final answer as the tool result.
…sage The model streams output audio continuously at real-time pace, silence included, rather than in per-response bursts. Emitting it as SpeechOutputAudioRawFrame lets the output transport derive BotStarted/StoppedSpeakingFrame from the audio itself; with TTS frames it would have counted the silence as speech and only stopped on the TTSStoppedFrame that follows the projected turn's end a couple of seconds later. Session usage arrives in the duration-based shape, which carries the delegated backend models' token counts rather than session token totals, so those are now reported as LLM usage metrics too, as deltas between the cumulative reports.
The aggregators' realtime mode writes the user message only once the assistant starts responding, to absorb transcripts that arrive late. With the Live API a user turn's final transcript arrives with its turn.done, and deferring the write placed a delegation's tool calls ahead of the user message that caused them in the recorded context. The service therefore no longer flags itself as a realtime service; it still recommends the external turn strategies, so user turns are proposed from the API's projected turns and written to the context as they end. reset_conversation() now drops the old session instead of closing it gracefully: the graceful close waits for the session's in-flight delegations to drain (up to the server's 10-second maximum), and their results belong to the conversation being replaced. An assistant turn the old session was in the middle of is closed out first so the aggregator records it and the response frames stay balanced.
A BackendLLMWorker job now has disjoint update and response semantics: updates carry the responses the backend gives before calling tools, and the job response carries its final answer. Before, the final answer was also sent as the last update, so a frontend that relayed updates as they arrived and delivered the job response — the cascade example's delegate tool — put the same answer into its context twice and ran the frontend LLM once per copy. OpenAILiveLLMService, which used to rely on the updates alone, now appends the returned final answer as speakable context after the job.
BackendLLMWorker sends the backend LLM's thought summaries as job updates of kind "thought", alongside the "text" kind for intermediate responses. OpenAILiveLLMService appends them to the delegation as commentary — the API's silent-context channel — so the live model knows what the backend is doing without speaking it, while intermediate and final responses stay speakable. The cascade example records them as developer messages without prompting a reply. The examples' Anthropic backends enable adaptive thinking with summarized display so the path is exercised.
… answered The async-tool guidance (ASYNC_TOOL_INSTRUCTIONS and the final-result message) now distinguishes the two ways a result reaches the model: while the user has an unanswered request, in which case the model answers it first and appends the result, and on a run of its own after the reply it landed behind has been spoken — the aggregator's normal deferred push — in which case the model delivers just the result without repeating its earlier reply. Replaying a cascade-frontend context where the joke had already been told: 8/8 samples re-told it under the previous wording, 0/8 under this one.
The updates a BackendLLMWorker streams while it works — what the backend says before calling tools, and its reasoning summaries — are progress, not answers. OpenAILiveLLMService appends both as commentary and only the final answer as speakable, the way the prompting guide uses the two channels. The cascade example records them as intermediate results of the delegate call without prompting a reply, so a progress note that lands alongside the answer is folded into one frontend run.
parse_server_event accepts the str | bytes the websocket yields, and the Responses-delegation reasoning config is narrowed with isinstance so its model_dump call type-checks.
realtime-openai-live-video.py demonstrates live video with OpenAIRealtimeLLMService, but next to the new realtime-openai-live-*.py examples the name reads as service "openai-live" + variant "video". The folder's other video examples (realtime-gemini-live-video.py, function-calling-openai-video.py) use plain -video as the variant name; follow suit.
The client-delegation and cascade examples seeded the backend's LLMContext with a system message; examples otherwise configure instructions through Settings.system_instruction. Move the backend instructions there, so the backend context carries only the tools and conversation — and the persistent-context restore no longer has to re-prepend them when it rebuilds the message list.
The guidance change now alters pipecat #5278's carefully tuned text as little as possible: the original first paragraph and the "never before your answer" ordering guard return, and the one sentence that unconditionally told the model to answer the user first now names both cases — answer first and append the result while something is unanswered, deliver just the result once nothing is. The final-result message mirrors the same two cases. Replaying the cascade repeat context against gpt-5.4-mini: 0/16 joke repeats (the unconditional wording: 8/8; the conditional prefix alone, without the already-answered case: 7/8). Re-running #5278's release-eval sweep — both async scenarios, four cascade bots, 20 runs each — passes 159/160 (that PR reported 154/160); the one failure is a judged aggregate that cut off mid-joke before the punchline arrived.
The delegate tool already logged the handoff; the updates streaming back and the final answer now log too, so a console run shows the whole frontend-backend exchange.
Everything a backend produces now leaves the worker as a BackendOutput — its text, whether it is a reasoning summary, whether it answers the task, and whether the user may hear it — sent as a job update. The final answer is one of those updates as well as the job response, so a frontend relaying output as it arrives can handle all of it in one place, while one that needs a return value keeps using the response. Responses are speakable and reasoning summaries are not. An app that wants a different split passes transform_output, which sees each output before it is sent and may rewrite its text or its speakability. A frontend whose model hands work over without wording a request can now omit the task and send only the conversation, leaving the backend to work out what is being asked.
The alpha's v3 contract changes the route, the startup handshake, every context-append event, the transcript stream, the delegation payload, function-call continuation and usage reporting, so OpenAILiveLLMService follows it across the board: - sessions start at /v1/live/sessions with a session.start message carrying the model, rather than a model query parameter and a session.update; later updates are sparse and carry delegation settings alone - turns are grouped here. The API emits timed transcript fragments and no turn events, so each direction accumulates fragments and closes a turn when its speaker falls quiet for transcript_turn_gap_secs. The two run independently: in full duplex both parties can hold a turn at once - a delegation names no task, so a client delegation hands the backend the transcript fragments since the last one and lets it work out the request - backend output is appended by its speakable flag: commentary for what the model should relay, thinking for what it should merely know. The words are the model's own vantage, which is why what crosses the worker boundary is flagged rather than named for a channel - a trailing developer message in the context becomes the documented speakable nudge once the session starts, so asking the bot to open the conversation works as it does with any other service - function results are queued as response items and the response is continued explicitly, once every call it made has been answered. The terminal snapshot's output list is empty by design, so the calls collected from the item events are what settles that - live usage is a cumulative duration; token metrics now come from the backend model's completed responses
The live model no longer words a request when it hands work over, so the client-mode prompts stop promising the backend a task and tell it to read the conversation instead; the frontend prompts drop the instruction to make each delegation self-contained, which the model now has no way to honour. The delegation handler logs the id it is given rather than text that no longer exists. A new example puts the backend in charge of what the user hears: it marks the lines it wants spoken, and transform_output turns that convention into the speakable flag. That is the same control the API's own reference client gets from a tool call, without the plumbing.
The backend worker was described and shown as if its frontend had to be a cascade pipeline, which is only one of the shapes it takes. A second example puts OpenAI Realtime in the frontend role, delegating through the same tool to the same worker, and the cascade example is renamed to say which of the two it is — picking up the parent-folder prefix the rest of examples/ uses along the way.
…l assistant text `speakable` (rather than always marking text `speakable` and thoughts not `speakable`).
The two-layer examples recorded every backend update as silent context regardless of its speakable flag, so an app that marked something for the user got no way to have it said. Running the LLM on the result is what gives a cascade or realtime pipeline a voice, the way the commentary channel does for a speech-to-speech frontend, so that is what the flag now drives. With only final answers speakable by default, the behaviour is unchanged until an app asks for more.
The Responses examples handed their backend gpt-5.4-mini, which the live model could nearly have handled itself. Delegation exists so the conversation can reach a heavier model than a live one can be — the backend thinks and calls tools while the live model keeps talking, spending that latency off the critical path — so they now name one, matching the delegation model the API's own quickstart uses.
Transcript deltas land on 200 ms frame boundaries, so recording each one as its own labelled line gave the backend a column of slivers to read through — "USER: Get", "USER: me the", "USER: weather in" — where a sentence belonged. Consecutive fragments from one speaker are now joined as they arrive; the deltas carry their own leading spaces, so concatenating them reproduces the original text.
The conversation handed to a backend was a list of loose dicts whose shape lived only in a docstring. It is now a list of TranscriptLine — a speaker and what they said — which is deliberately narrower than a context message: the backend reads labelled transcript text, so there is no place in it for tool calls, images or multi-part content, all of which the renderer would have dropped without saying so. The job's request is typed the way its updates already were. The parameter is 'conversation' rather than 'messages', since these are not messages the backend's LLM will see: they are rendered into one.
A backend request now carries standard context messages, so a frontend can hand over a slice of its own context unchanged rather than converting it first. They are still flattened into one labelled transcript, which is what keeps the two conversations apart: the backend's own context holds only what the backend said, so it never mistakes the frontend's speech for its own. Only what a transcript can hold survives that flattening — user and assistant text. Tool calls, results and non-text content are skipped, and say so at debug level rather than vanishing quietly.
The transcript's text-only rendering read as a rule of the contract; it is a simplification, and carrying images or tool results is a question of how to render them rather than something ruled out. The conversation parameter now says what it holds — the conversation the user is having with the frontend — instead of the vaguer 'what has been said', and notes that a caller normally sends only what is new because the backend's own context keeps the rest.
A backend request is now just text the frontend composed, so the worker no longer decides how a handoff is worded. That was two fields and a branch — a task, a conversation, and a rule about what to render when — for something the frontend is better placed to settle: it knows whether its model worded a request or merely signalled a handoff. render_transcript_request() renders a conversation as a labelled transcript for the common case, with the instruction that follows it as an argument, so wording a request is now what that argument is for rather than a separate path through the renderer. An application with its own frontend-to-backend protocol can skip the helper. The roll-your-own examples take the handoff route the Live API's own clients take: their delegate tool words nothing, passing the conversation the frontend has had since the backend last saw it.
The transcript header dated itself from the backend's last 'task', a word the API surface no longer uses and one the two layers never agreed on. The delegation is what the boundary actually is, and it is what the Live API's own reference client calls it.
Rebooking a cancelled flight runs three dependent tool calls over about ten seconds, so the backend has both working-out to keep to itself and news the user is waiting on.
The rebooking scenario needs a flight number to get going, and the tools take any.
The example turns on saying the right thing to set the backend going, and the terminal is where someone is looking as they join.
It arrives in a stream of debug lines, so it is coloured and flagged.
A frontend weighs `prefers_spoken` rather than obeying it: the live model may speak an output marked otherwise, or skip one marked this way, as the conversation calls for.
`OpenAILiveLLMService` is what reads `prefers_spoken` here, and what it does with the flag is its own business.
The live service and the backend worker land as separate entries, and the example that shows a backend choosing what the user hears is named apart from the two that show the delegation modes.
A teardown that fails part-way clears `_disconnecting` anyway, so the next session can still send. The job response carries the answer as transformed, matching the final update the docstring says it duplicates. A payload with no speech flag rebuilds with the field's default rather than silencing the output. Also covers the tool-set frame, which is what prompts a session update when a continuous session sees no context frame.
A flag the payload omits is left off the constructor call, so a default is declared once; one it carries is still coerced, since a payload can cross a bus from another process.
A late error while a session shuts down is no longer reported as a startup failure, which ends the pipeline under the examples' unusable policy. The model is told delegated work failed without the exception text, which it would otherwise paraphrase aloud; the detail goes to the error. The model and turn gap are read when a session is configured, so a change applied mid-session takes effect on the next one, as the warning says. Tests record what goes on the wire, so the append events' required null delegation_id is covered.
Until a session has started on the current connection an error is that session failing to start; afterwards it belongs to one that is running or shutting down. A reset opens a new connection, whose startup can fail in turn.
The gap groups transcript fragments in this service, not at the API, so a change to it need not wait for a new session. What the API does fix at session start keeps its own warning, and the rest uses the shared unhandled-settings warning.
Nothing has needed to vary it per session, and a setting the API cannot carry is one more thing to explain.
A setting the API fixes at session start is one this service does not handle at runtime, which is what the shared warning is for.
A turn's open and close hold the turn's lock, so the gap timer, which runs in its own task, can no longer emit one turn's end frames among the next turn's start frames. Delegated Responses events correlate on the envelope's delegation id alone. The wrapped events do not all name their response, so an envelope that arrives without a delegation id falls back to a shared key rather than splitting one response across several. A backend LLM error ends the delegation and answers the job with JobStatus.ERROR, and a delegation that produced no text tells the live model so: either way the model gets a word back and the conversation moves on. A tool handler that raises is left alone, since the LLM service settles the call and carries on. Context appends are chunked by estimated tokens, counting characters outside ASCII whole, so CJK text stays within the API's 500-token part limit. The tools comparison per user turn derives only the tool configuration, leaving the startup history — which grows with the conversation — unbuilt.
An LLMSetToolsFrame is what announces a new tool set, and session.started is when the API will take one, so those are the two places a sparse session update goes out. That leaves _handle_context to configure the first session and take the context, matching the sibling realtime services, and it drops the tools comparison from the per-turn path.
The model is gpt-live-1 and the session is opened with the bearer token alone. Server events are still parsed leniently, so an event type this module doesn't model becomes UnknownServerEvent rather than an error.
3 tasks
Codecov Report❌ Patch coverage is
... and 47 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
GPT Live (
gpt-live-1) is OpenAI's full-duplex speech-to-speech model: it listens and speaks at the same time, decides on its own when to answer and when to stop when talked over, and needs no client-side VAD, turn detection, orresponse.create. The pipeline continually streams audio in and plays audio out.It works in a two-layer architecture:
The Live API offers two ways to run the backend — its two delegation modes:
This PR adds
OpenAILiveLLMServicewith both modes supported, running the client-delegation backend on a reusable piece —BackendLLMWorker, which wraps any Pipecat LLM service with its own context and tool loop.Terminology used throughout the code and docs: frontend and backend are the two layers' roles; the live model is GPT Live specifically, i.e. the frontend in this service.
What's added
OpenAILiveLLMServiceand supporting filesBackendLLMWorker+BackendOutput, for defining and wiring up the backend LLM that client delegation hands work to. Everything the backend produces comes back as aBackendOutputsaying what it is and whether the user may hear it;transform_outputdecides that per output, or rewrites the text on its way outrealtime-openai-live-video.pyexample (OpenAI Realtime + live video) torealtime-openai-video.py, so it no longer reads as an OpenAI Live example — matching the plain-videovariant naming ofrealtime-gemini-live-video.pyandfunction-calling-openai-video.pyUsage
Responses delegation — the backend's settings are the same
OpenAIResponsesLLMSettingsthe Responses service takes; its tools (and their handlers) come from the pipeline'sLLMContext:Client delegation — any Pipecat LLM service as the backend, with its own context and tools:
How it maps onto Pipecat
session.starton the firstLLMContextFrame(the app's kickoffLLMRunFrame): a leading system message orSettings.system_instructionbecomes the instructions, the remaining text messages seed it as prior conversation. A trailing developer message becomes the documented speakable nudge once the session starts, so asking the bot to open the conversation works as it does with any other service.EndFramecloses gracefully;reset_conversation()restarts from the current context (persisted-context flows).SpeechOutputAudioRawFrame, because the model streams continuously at real-time pace, silence included, and the output transport derives bot-speaking state from the audio itself.TURN_GAP_SECS(0.8), the two running independently because in full duplex both parties can hold a turn at once. Assistant turns →LLMFullResponseStart/EndwithLLMTextFrame/TTSTextFramedeltas; user turns → interim/finalTranscriptionFrames plusProposedUserStarted/StoppedSpeakingFrame, resolved by the recommendedExternalUserTurnStrategies(enable_interruptions=False). Only the pipeline frames need the grouping; the delegation payload ships raw fragments, which the guide prefers for prompt construction.InterruptionFrameis never broadcast and every tool behaves ascancel_on_interruption=False(cancellable_by_llmremains the way to let the model stop a tool on request).run_function_calls; each result is queued as a response item and the response continued explicitly, once every call it made has been answered. The terminal lifecycle snapshot'soutputlist is empty by design, so the calls collected from the individual item events are what settles that. Results still go back as the frames pass through the service, not via the context round-trip the other realtime services use (see Follow-ups).runjob carrying the transcript fragments since the previous delegation, and the backend works out the request from them. What comes back is appended by itsprefers_spokenflag: commentary for what the model should relay in its own words, thinking for what it should merely know. The API's two channel names describe what the live model does with the text — it is appending to its own reasoning stream — which is why what crosses the worker boundary is flagged rather than named for a channel: ordinary backend prose can be unspeakable without being anything like a thought. A run is finished when every LLM run it triggered has ended and no tool call is in flight, counted as the LLM picks up context frames (exact even when a tool returns before its response ends).session.usage.updatedis logged as cumulative seconds; token metrics come from the backend model's completed responses instead.Where the API isn't explicit, and what we do
BackendLLMWorker's job issequential=True.request.response.output_item.doneonly —response.function_call_arguments.donelacks the call id and name.strict— the session schema rejects unknown fields, andstrictis Responses-only.session.startedechoes less than the docs show and unknown event types may appear as the API grows; unrecognised events are logged, not fatal.transform_outputgives an app per-utterance control over what the user hears.Validation
Automated. 60 unit tests pass, covering session startup for both modes, transcript fragments → turn frames (including a gap closing a turn and both speakers holding one at once), the batched function-call continuation, client-delegation dispatch with the new append events, the
prefers_spokenbaseline and atransform_outputthat rewrites text, and the backend worker's tool loop (fast-tool race included).ruffandpyrightare clean.Against the API. Validated end to end:
LLMRunFrame)commentaryis the spoken channel,thinkingthe silent oneload_conversationdeliberately never callsresult_callback, which now has to coexist with the explicit response continuationAgainst GA (2026-09-10). With
gpt-live-1and the bearer token alone: the Responses-delegation example passes the release-evalsscripted/weather_function_call_audioscenario (greeting,get_current_weathercall, spoken answer; 22.6 s). The client-delegation example delegates, and the backend's answer is spoken 12 s after the delegation ("Anyway, it's currently 75 degrees and nice in Washington, DC"); thelive_joke_while_waitingscenario's observe turn misses it only because the answer lands in the same assistant stretch as the joke.Open TODOs
DEFAULT_MODELisgpt-live-1and the session opens with the bearer token alone.TURN_GAP_SECSconstant rather than a setting — nothing needs to vary it per session yet. There is no prescribed value to adopt. The value is what the turn groupers use, so it decides what the context records, and that reads correctly in the eval runs. Measured over one session: within-speaker gaps were 200 ms throughout bar three at 600 ms, leaving 200 ms of headroom — but that is one scripted TTS voice, so it is a floor, not a distribution. Human thinking pauses, other languages, or backchannel handling (which we don't implement) would be the reasons to revisit; we do have the mic audio, so local VAD could supply audio-presence signals the API doesn't expose.MAX_CONTEXT_APPEND_TOKENS = 450, measured per character — four ASCII characters to a token, one for anything else — so a CJK answer, which runs about a token per character, chunks as tightly as an English one.prefers_spokenbaseline: only the final answer, decided by listening. With progress speakable the backend's filler arrives while the frontend is still mid-sentence and talks over it, stacking two acknowledgements ("Sure thing" / "Of course"). Silencing progress fixes it, and matches the guide's ownthinkingexample, which is a progress sentence.start_ms(what SRT's ordering buys) would also change nothing: across 53 fragments, arrival order matchedstart_msorder exactly, with no cross-speaker overlap. And the renderer takesLLMStandardMessage(role + content), so carrying timings would mean pollutingcontentor forking it.session.delegation.createdhas nocontentfield in either mode, and the Responses form adds onlyresponse_id, whichDelegationMetadatamodels.session.delegation.createdand left both to the backend, which calledget_current_weatherandget_restaurant_recommendationwithin that one delegation (its own reasoning: "The user wants two things simultaneously"). Confirmed by hand too. If two ever do overlap, the second finds the transcript ledger drained and gets the instruction line alone — though the backend still has its own context to read, andrunissequential=True, so delegations queue rather than race.session.started, and a trailing developer message is delivered that way now.Deliberately not in this PR
CLI
pipecat initregistry entries; runtime context injection viaLLMMessagesAppendFrame(punted like the sibling services, with a code-comment note on the unambiguoussystem/developercase); WebRTC and the JSON session-creation route, sideband (/attach) and SIP, microphone mute/unmute, the 8/16 kHz telephony formats,context_management,session.instructions.appendas a public API, runtime delegation updates, custom voices, per-delegation context forks.Follow-ups (separate PRs)
session.closedreasons distinctly —close_requested,expired,content,remote_hangup,connection_lost— where today every close gets the treatment of one we asked for. And bring error handling in line with the other WebSocket services: back off and retry a transient failure, push the error withforce_treat_as_permanenton the last attempt, and push a permanent one (rejected credentials, a malformed request) straight away, judging permanence from the HTTP/WebSocket status when the API gives nothing better. Draft PR to be linked here.ClientDelegation.backendname a worker registered elsewhere. Today it takes a worker object, which the service adds as a child of the pipeline worker insetup(), so the backend can only run in the same process. A backend on another machine over aRedisBus/PgmqBusis a name the frontend resolves through the registry, never an object it holds, andjob()already addresses workers by name. Widen the field toBackendLLMWorker | str: an object is added as a child as now; a name is only addressed, and the app registers the worker itself (locally, or in the backend's own process with the shared bus). Add TwoLayerLLMService: a roll-your-own two-layer bot with any frontend #5689 gives the roll-your-own stack this samebackend=shape and is the pattern to follow, tests included.speak_to_user-style tool on the backend, so the backend model chooses per utterance instead oftransform_outputreading a marker convention after the fact. An app can already do this with no framework changes: a backend tool reaches its worker throughparams.pipeline_worker, takes the delegation's id from the publicactive_jobs(exactly one, sincerunissequential=True), and sendsBackendOutput(text=..., prefers_spoken=True).to_payload()as a job update. Measured against Claude with a slow lookup: two spoken lines reached the frontend — one before the lookup, one with the answer — and the final response came back empty, which is the shape of a backend that speaks through a tool. What a follow-up would add is folding it into the worker so those updates run through_emitand pick uptransform_output, which a hand-rolled tool bypasses.LLMContextFrame. In realtime mode the aggregator never re-pushes a result that lands while the user is speaking, so the provider never gets the output. Consider migrating them to the direct path used here.OpenAIResponsesLLMServicesilently ignores settings the Responses API doesn't take (frequency_penalty,presence_penalty,seed,top_k,max_tokens); warn as this service does.LLMContextSummarizationUtil.estimate_tokens()(utils/context/llm_context_summarization.py) countslen(text) // 4whatever the script, so a CJK conversation reads as a quarter of its real size and auto-summarization fires far too late. Making it script-aware — CJK runs about a token per character — would fix that and let this service's local_estimated_tokensgo: the two are the same heuristic and agree exactly on ASCII. Left out here because it changes summarization thresholds for every pipeline, which belongs in its own PR.Reporting the backend's tool calls over RTVI
What an RTVI client sees today:
self.run_function_calls(...), in the main pipelineA later PR only fills the empty cell, so it breaks nobody adopting this one: no message emitted today changes shape or stops being emitted, since client delegation reports no tool calls at all.
The backend's frames still have to reach the main pipeline. Two existing routes, neither of them new public API:
PipelineWorker(bridged=...)gives the backend worker bus edge processors so its frames can cross, or the calls ride the job channel and the frontend re-emits them — which means carrying call metadata alongsideBackendOutput, the one genuinely additive change, and the same one thespeak_to_userfollow-up wants.