Design GPT-Live as a voice intermediary for multiple Gromozeka agents

[Lewik]

Problem and intended experience

Gromozeka should support a continuous voice conversation that lets the user operate existing agents, inspect their progress, and hear their results while continuing to use the visual interface. The working agents retain their own conversations, identities, reasoning, tools, and memory.

The product analogy is a robot operating an existing mechanical gearbox: GPT-Live 1 provides a conversational control layer over Gromozeka's existing controls. It does not own a replacement agent runtime.

This issue preserves the design discussion and research from September 12–13, 2026. It is a design checkpoint, not an implementation-ready specification. No Live integration or authenticated Live API experiment has been completed.

Related: #54 (hands-free voice interaction) and #58 (streaming transcription with gpt-live-transcribe). This proposal explores a separate, more conversational mode using gpt-live-1; it does not supersede the classic STT/TTS or transcription-only paths.

Product direction from the discussion

• Keep substantial reasoning and execution in Gromozeka agents. The user's preferred main reasoning models are GPT-5.6/GPT-6 with maximum available reasoning effort; the architecture should also accommodate other providers.
• Give Live a distinct, stable identity as the voice intermediary. The user should perceive Live and the working agents as separate entities: “Architect reports…”, “DevOps is still working…”. Switching the addressed backend should not require Live to impersonate that backend.
• Live handles listening, interruptions, acknowledgments, short clarifications, natural phrasing, and delivery of prepared results. Requests needing substantive reasoning or action should reach a working agent.
• Support addressing several agents through one voice session, including while earlier work continues. Preserve the exact recipient, conversation, request, and source message.
• Use both visual and audio feedback. The UI exposes pending requests and actual task state; Live receives a concise description of what is currently shown and selected.
• Prefer forwarding the backend's existing ttsText/voice section to Live. Do not introduce another model merely to rewrite every outgoing agent response. Full visual answers remain available in the UI.
• Preserve the distinction between stopping speech, cancelling a task, correcting an existing request, and issuing a new request. A repeated command must not implicitly mean “cancel the previous operation.”

Proposed flow

User speech <-> Live voice conversation
                   |
             transcripts + delegation signal
                   v
UI state + relevant chat context -> application-owned intent interpretation
                                             |
                                      addressed operation(s)
                                             v
                                existing Gromozeka runtime
                                             |
                                agent results and task state
                                  /                      \
                         original UI messages      author metadata + ttsText
                                                           |
                                                           v
                                                          Live

Incoming requests

The interpretation layer receives the recent Live conversation, relevant working-agent chat context, the current UI selection, and ongoing task state. It determines the intended recipient and whether the request creates work, supplies more information, corrects a request, or cancels a particular operation.

Using a reasoning model here is a serious candidate, especially for references spanning both the voice conversation and a backend chat. Its exact model, context construction, and ownership remain open. It need not appear as a third user-facing persona.

An important alternative is to process accumulated transcript text independently of Live's delegation decision. OpenAI explicitly supports application-owned work triggered from transcripts. This can reduce dependence on Live deciding when to delegate. Coordinate transcript-triggered work and later delegation events through the same recorded user intent to prevent duplicate execution. A fragment is not necessarily a complete request; subsequent speech can correct it.

Dispatch through the existing runtime operations for posting messages, invoking agents, queuing input, and controlling execution. Keep the existing safe insertion points after tool results and at turn completion.

Outgoing results and attribution

Preserve the original backend result with its author and message ID. Send prepared speech content directly to Live with concise source/task metadata. Use application logic to order concurrent updates and suppress obsolete or duplicate deliveries.

A proposed internal envelope could carry conversationId, sourceAgentId, sourceMessageId, operationId, revision, kind, and ttsText. These are design candidates for Gromozeka, not native Live API fields. The public append API receives plain text, so the bridge must render a short factual description of the source and result.

Keep Live's transcript separate from the original agent message. Do not feed Live's own acknowledgments or spoken paraphrases back as new user commands. UI presentation can distinguish the original authored result from Live's presentation of it.

Interface awareness

Generate context from application state: current conversation, selected message, visible agents/tasks, pending requests, and relevant status changes. Send an initial snapshot and concise changes through session.thinking.append with delegation_id: null. Explicitly describe corrections to prior selections; coalesce rapid changes and omit unchanged updates.

Do not assume that a visible message has been read. A UI snapshot should describe what is displayed, not invent user awareness. Once an utterance has been routed, later tab changes must not silently retarget it.

For Gromozeka's own UI, no screenshot-analysis model is needed to construct this context. Image/screen understanding beyond known application state is a separate capability to design; the inspected Live frontend protocol does not provide general image input.

Verified public API constraints from the research

• Client delegation fits this design: the application chooses its own backends, multiple models, routing, and context. Native delegation does not provide a list of Gromozeka agents or select one by ID.
• session.delegation.created supplies a delegation ID, target, and timing metadata, but no task text or tool arguments. Construct the request from transcript events and application state. Do not assume one delegation corresponds to exactly one application operation.
• session.input_transcript.delta and session.output_transcript.delta are timed fragments, not authoritative complete turns. Keep sufficient history for corrections and short replies.
• session.thinking.append supplies quiet factual context; session.commentary.append supplies information intended for speech, which Live may paraphrase; session.instructions.append adds trusted behavior guidance. Each public append accepts up to 500 tokens. Use a known client delegation ID for associated updates or null for session-wide context.
• Append acknowledgment confirms acceptance, not that the user has heard the information or that an external operation completed. Quiet context can influence later speech; it is not a guarantee of silence or secrecy.
• The output-transcript schema has an optional client_event_id, but it does not establish exact semantic provenance for every spoken word. Distinct personas and source labels help; authoritative attribution must remain in application records.
• Prompting can encourage delegation and a narrow conversational role. It does not turn Live into deterministic verbatim TTS or guarantee that it never speaks on its own. Timing, paraphrasing, backchannels, and interruption behavior need evaluation.
• Stopping local playback, stopping Live's speech generation, and cancelling backend execution are different operations. Live does not use the classic Realtime response.create voice-turn loop; do not carry over incompatible control assumptions.
• Public transport options include WebSocket and WebRTC with a server-side sideband connection. Transport, session recovery, context restoration, and credential ownership still need a concrete Gromozeka design.

Subscription path: separate research track

The inspected Codex Rust source uses an internal gpt-live-1-codex path and an OAuth/WebRTC call flow. It also contains private framing such as delegation.context.append and session.context.append; its internal handoff event can carry request text. These details differ from the public Live event schema above.

Do not treat public API keys and subscription OAuth as interchangeable, or assume the existence of this source code proves entitlement or successful third-party access. The inspected standalone audio WebSocket and OAuth/WebRTC paths also have different authentication requirements. No authenticated request was made during this research.

The latest inspected Codex revision was 7efa9d96fb34c3cafe108a3c870bfc33e5635772. Recheck the source and supported access path before implementation. Its current handoff routing targets the same Codex session; it is not already a multi-Gromozeka-agent router.

Existing Gromozeka building blocks

These observations were checked in the local b646da6edd96e10efe1b6b1307bde796f7358595 snapshot, not a claim about current remote main:

• ConversationRuntimeService: explicit agent invocation, message posting, queue placement, cancellation, and execution control.
• Conversation: participants, message authorship, replyTo, and structured fullText/ttsText/voiceTone.
• ClientPresentationRegistry: already selects ttsText for speech presentation.
• VoiceInputDelivery: stable target capture at voice-input start must be preserved when designing broader session scope.
• AgentMention: multiple mentions in one message are currently rejected. Multi-agent voice dispatch is additional behavior to design.

Useful external implementations

• Pipecat spoken-update example: an Anthropic backend marks which outputs should be spoken; transform_output maps the marker to prefers_spoken. This closely matches Gromozeka's TTS section. BackendOutput explicitly treats the preference as a hint, not guaranteed playback. Merged integration PR.
• LiveKit client delegation: combines maintained chat context with a pending transcript to interpret a delegation. pending_transcript is a framework convenience, not task text supplied by the native public event. Some plugin documentation still reflects alpha-era constraints; use current OpenAI documentation for the API contract.
• Codex realtime history: keeps voice transcript segments and references to original backend items separately. This is useful for preserving authorship without duplicating the original result into a new ordinary message.

Open decisions and evaluation cases

• Choose session scope: one conversation with multiple agents, or a voice session spanning an explicitly selected set of conversations/workspace tasks. Define how explicit recipient names interact with current UI selection.
• Choose intent-interpreter ownership/model and how much backend history it reads. Evaluate delegation-triggered and transcript-triggered dispatch rather than assuming either is sufficient.
• Define durable mappings between utterances, interpreted intents, delegation IDs, operations, source messages, and delivery state; distinguish intentional repetition from transport/event duplication.
• Define concurrency, ordering, corrections, cancellation, reconnect, and stale-result behavior. Existing execution policies still apply: displaying two release requests does not authorize two concurrent deployments.
• Define when Live may answer directly, how it names sources, and how spontaneous speech is represented in the UI. Test whether the resulting behavior is sufficiently predictable compared with classic STT/TTS.
• Select and verify the supported public API or subscription access path; do not assume protocol parity.
• Evaluate: two agents addressed in one utterance; a correction during a tool call; “cancel the previous one”; a tab switch while speaking; simultaneous backend results; reconnect after work started; barge-in that only stops speech; and clear separation of Live's own words from backend-authored results.

Primary API references

• Delegation mode and context ownership
• Maintaining conversation context
• Application work triggered from transcripts
• Sharing UI context
• Appending instructions, context, and commentary
• Output transcript event schema

Research evidence is dated. Refresh the relevant contracts and source before turning this checkpoint into an implementation plan.