Skip to content

feat(phonic): handle user text input via generate_reply - #5

Draft
devin-ai-integration[bot] wants to merge 7 commits into
mainfrom
devin/1781807418-phonic-text-input
Draft

feat(phonic): handle user text input via generate_reply#5
devin-ai-integration[bot] wants to merge 7 commits into
mainfrom
devin/1781807418-phonic-text-input

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Description

The Phonic realtime plugin currently only handles audio input from users. When a user types text (e.g. via LiveKit's chat/text stream), the framework calls updateChatCtx with the user message followed by generateReply. Previously, the plugin silently ignored user messages in updateChatCtx, so typed text input was dropped with no response.

This PR adds text input support by:

  1. Detecting new user messages in updateChatCtx and storing the text as pending
  2. Including the pending user text in the generate_reply message's system_message field, so the downstream echo service sees the user's typed input and the LLM responds to it

Changes Made

  • Added pendingUserText field to RealtimeSession to buffer user text between updateChatCtx and generateReply
  • In updateChatCtx: detect items with role === 'user' and store their text content
  • In sendGenerateReply: if pending user text exists, format it into the system_message so the downstream LLM context includes the user's typed message
  • Removed spurious warning when updateChatCtx is called with only user messages

Pre-Review Checklist

  • Build passes: All builds (lint, typecheck, tests) pass locally
  • AI-generated code reviewed: Removed unnecessary comments and ensured code quality
  • Changes explained: All changes are properly documented and justified above
  • Scope appropriate: All changes relate to the PR title

Testing

  • Full monorepo pnpm build passes
  • ESLint passes on the phonic plugin
  • Prettier formatting check passes

Additional Notes

The flow for text input with this change:

User types text → LiveKit room text stream
  → DEFAULT_TEXT_INPUT_CALLBACK → sess.generateReply({ userInput: text })
    → AgentActivity.realtimeReplyTask
      → updateChatCtx (stores pendingUserText)
      → generateReply → sendGenerateReply
        → socket.sendGenerateReply({ type: 'generate_reply', system_message: 'The user sent... "text"...' })
          → echo downstream sees user text in system_message, LLM responds

Link to Devin session: https://app.devin.ai/sessions/51df28f166ac4fff8cae41b88dc9bab2
Requested by: @qionghuang6

When a user sends text (e.g. typing in a chat), the framework calls
updateChatCtx with the user message followed by generateReply.
Previously the Phonic plugin ignored user messages in updateChatCtx,
so text input was silently dropped.

This change:
- Detects new user messages in updateChatCtx and stores the text
- Includes the user text in the generate_reply system_message sent
  to the Phonic downstream service, so the LLM can see and respond
  to typed user input

Co-Authored-By: qiong@phonic.co <qionghuang6@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

qionghuang6 and others added 6 commits June 18, 2026 15:16
Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>
…onic-text-input

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	plugins/phonic/src/realtime/realtime_model.ts
- Demote "Received user text input" log from info to debug
- Remove leftover agentName from the realtime example

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants