Feature Type
I cannot use LiveKit without it
Feature Description
According to deepwiki https://deepwiki.com/search/can-fallbackrealtimesession-im_15e6ca5e-6280-4091-b3d6-65ddfb72aedf?mode=deep,
for livekit.agents.llm.RealtimeModelAapteri, openai specific realtime session event handlers are attached to its child realtime session and handlers for general reatime session event in
|
_FORWARDED_EVENTS: tuple[EventTypes, ...] = ( |
) are forwarded to attached to its
parent realtime session.
When model fallback happens, its old child realtime session are closed and discarded immediately before the new one is created and its parent realtime session sill keep those handlers of those general realtime session event. As a result, those openai specific realtime session event handlers original attached to the child session will lose. Therefore, we need to resubscribe those openai specific realtime session event handlers to the new child realtime session. It has to be done manually every time when we have openai specific realtime session event handlers to attach.
Can you modify the source codes so that the resubscribe is implemented automatically when realtime model is fallbacked to the same plugin-type model (eg. openai -> openai, google -> google)?
Workarounds / Alternatives
Since we have set realtime model fallback from openai to Azure openai as below:
llm=RealtimeModelFallbackAdapter(
[openai.realtime.RealtimeModel(),
openai.realtime.RealtimeModel.with_azure()]
)
,
we need to manually resubscribe those openai specific realtime session event handlers to the new child realtime session through the codes like below:
def attach_oai_realtime_event_handlers(self, rt_session: RealtimeSession) -> None:
if not isinstance(rt_session, livekit.plugins.openai.realtime.realtime_model.RealtimeSession:
raise TypeError(f"Expected OpenAI RealtimeSession, got {type(rt_session).__name__}")
@rt_session.on("openai_server_event_received")
def on_openai_server_event_received(event: dict[str, Any]) -> None:
pass
if type(rt_session).__name__ == "_FallbackRealtimeSession":
current_child_session = rt_session._active
@rt_session.on("session_reconnected")
def on_session_reconnected(_: RealtimeSessionReconnectedEvent) -> None:
nonlocal current_child_session
new_child_session = rt_session._active
# check whether the new child session is different from the current child session
# to make sure the received 'session_reconnected' event is emitted by model fallback
if new_child_session is not current_child_session:
LOGGER.info(
"Resubscribe OpenAI realtime event handlers when model fallback happens "
"for RealtimeFallbackAdapter"
)
attach_oai_realtime_event_handlers(new_child_session)
current_child_session = new_child_session
Additional Context
No response
Feature Type
I cannot use LiveKit without it
Feature Description
According to deepwiki https://deepwiki.com/search/can-fallbackrealtimesession-im_15e6ca5e-6280-4091-b3d6-65ddfb72aedf?mode=deep,
for
livekit.agents.llm.RealtimeModelAapteri, openai specific realtime session event handlers are attached to its child realtime session and handlers for general reatime session event inagents/livekit-agents/livekit/agents/llm/realtime_fallback_adapter.py
Line 58 in 87ef1b1
When model fallback happens, its old child realtime session are closed and discarded immediately before the new one is created and its parent realtime session sill keep those handlers of those general realtime session event. As a result, those openai specific realtime session event handlers original attached to the child session will lose. Therefore, we need to resubscribe those openai specific realtime session event handlers to the new child realtime session. It has to be done manually every time when we have openai specific realtime session event handlers to attach.
Can you modify the source codes so that the resubscribe is implemented automatically when realtime model is fallbacked to the same plugin-type model (eg. openai -> openai, google -> google)?
Workarounds / Alternatives
Since we have set realtime model fallback from openai to Azure openai as below:
,
we need to manually resubscribe those openai specific realtime session event handlers to the new child realtime session through the codes like below:
Additional Context
No response