Add client-side tool calling to push-to-talk example

[jamsea]

What

Shows how to run an LLM tool call in the browser instead of on the server, using the current RTVI flow rather than the deprecated RTVIProcessor.handle_function_call.

The example tool is get_browser_info. Ask the bot "what browser am I using?" and it reads back your real user agent, screen size, language, and time zone, none of which the server ever saw.

How it works

1. The server advertises the tool in LLMContext(tools=...) and registers run_on_client as its handler.
2. The LLM calls it. Pipecat tells the client via llm-function-call-in-progress.
3. run_on_client returns without calling params.result_callback, which leaves the call open instead of answering it.
4. The browser handler runs and returns a value, sent back as llm-function-call-result.
5. Pipecat turns that into the call's result, adds it to context, and reruns the LLM.

Two halves: run_on_client plus the CLIENT_TOOL wiring in server/bot.py, and client/src/app/hooks/useClientTools.ts.

Why push-to-talk

• It already demos client to server messaging via sendClientMessage, so this completes the two-way picture in one example.
• It has a single client, which matters here. A tool the client never answers leaves the call open for the rest of the session, because function_call_timeout_secs does not cover deferred calls (the timeout task is cancelled as soon as the handler returns). Adding this to a multi-client example would hang the bot for every client that did not get the handler.

Note on the report level

Client-run tools must opt in to FULL. function_call_report_level defaults to NONE, which sends only a tool_call_id, and a client cannot dispatch a call it cannot name. Other tools stay at the secure default.

Verified end to end

Run against a live Daily session on 2026-08-12. Pipecat 1.7.0, @pipecat-ai/client-js 1.13.0, client-react 1.8.1, daily-transport 1.6.8, voice-ui-kit 0.13.0. Asked the bot out loud for browser info:

12:10:26.305  Calling function [get_browser_info:call_NJ8...] with arguments {}
12:10:26.306  Delegating get_browser_info to the client        <- no-op handler, call left open
12:10:26.310  FunctionCallInProgressFrame                       <- client told
12:10:26.558  FunctionCallResultFrame                           <- client answered, 253 ms

The LLM then reran with the browser's real data in context, which the server never had:

{"language": "en-US", "screen": "1728x1117", "timeZone": "Asia/Taipei",
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... Chrome/151.0.0.0 Safari/537.36"}

Note arguments {}: the tool takes no inputs, which is why properties is empty.

Barge-in while the bot is speaking also works: five clean interruptions in the same session, each starting a fresh user turn.

[Copilot]

Pull request overview

Adds a push-to-talk example enhancement that demonstrates client-side LLM tool calling (browser-executed tool logic) using the current RTVI flow, along with documentation and dependency updates to support the new client tooling.

Changes:

• Advertise and register a get_browser_info tool on the server, while delegating execution to the browser via RTVI function-call frames.
• Add a client hook to handle the tool call in-browser and return browser-derived data to the bot.
• Update docs and client dependencies to reflect the new capability.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
README.md	Updates the top-level examples list to mention client-side tool calling in push-to-talk.
push-to-talk/server/bot.py	Defines the get_browser_info tool schema, registers a no-op server handler, and configures RTVI function call reporting for client dispatch.
push-to-talk/README.md	Documents the new client-side tool calling behavior and use case.
push-to-talk/client/src/app/hooks/useClientTools.ts	Adds browser-side tool handler wiring for get_browser_info.
push-to-talk/client/src/app/components/App.tsx	Initializes the client-side tool handlers within the app.
push-to-talk/client/package.json	Updates/aligns Pipecat client and UI kit dependencies required for the example.
push-to-talk/client/package-lock.json	Locks updated dependency versions and adds transitive deps introduced by the upgrades.

Files not reviewed (1)

• push-to-talk/client/package-lock.json: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.