Skip to content

Should automatically resubscribe those plugin specific rt_session event handlers to new child rt_session when realtime model is fallbacked to the same plugin-type model #6556

Description

@moz164164

Feature Type

I cannot use LiveKit without it

Feature Description

According to deepwiki https://deepwiki.com/search/can-fallbackrealtimesession-im_15e6ca5e-6280-4091-b3d6-65ddfb72aedf?mode=deep,
for livekit.agents.llm.RealtimeModelAapteri, openai specific realtime session event handlers are attached to its child realtime session and handlers for general reatime session event in

_FORWARDED_EVENTS: tuple[EventTypes, ...] = (
) are forwarded to attached to its parent realtime session.

When model fallback happens, its old child realtime session are closed and discarded immediately before the new one is created and its parent realtime session sill keep those handlers of those general realtime session event. As a result, those openai specific realtime session event handlers original attached to the child session will lose. Therefore, we need to resubscribe those openai specific realtime session event handlers to the new child realtime session. It has to be done manually every time when we have openai specific realtime session event handlers to attach.

Can you modify the source codes so that the resubscribe is implemented automatically when realtime model is fallbacked to the same plugin-type model (eg. openai -> openai, google -> google)?

Workarounds / Alternatives

Since we have set realtime model fallback from openai to Azure openai as below:

llm=RealtimeModelFallbackAdapter(
 [openai.realtime.RealtimeModel(),
openai.realtime.RealtimeModel.with_azure()]
)

,
we need to manually resubscribe those openai specific realtime session event handlers to the new child realtime session through the codes like below:

def attach_oai_realtime_event_handlers(self, rt_session: RealtimeSession) -> None:
  if not isinstance(rt_session, livekit.plugins.openai.realtime.realtime_model.RealtimeSession:
     raise TypeError(f"Expected OpenAI RealtimeSession, got {type(rt_session).__name__}")

    @rt_session.on("openai_server_event_received")
    def on_openai_server_event_received(event: dict[str, Any]) -> None:
          pass

if type(rt_session).__name__ == "_FallbackRealtimeSession":
  current_child_session = rt_session._active
  @rt_session.on("session_reconnected")
  def on_session_reconnected(_: RealtimeSessionReconnectedEvent) -> None:
      nonlocal current_child_session
      new_child_session = rt_session._active 
      # check whether the new child session is different from the current child session
      # to make sure the received 'session_reconnected' event is emitted by model fallback
      if new_child_session is not current_child_session:
          LOGGER.info(
              "Resubscribe OpenAI realtime event handlers when model fallback happens "
              "for RealtimeFallbackAdapter"
          )
          attach_oai_realtime_event_handlers(new_child_session)
          current_child_session = new_child_session

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions