Skip to content

[LLM:Feature] Add Agentic Flow to MNN LLM Chat#4512

Open
huangzhengxiang wants to merge 6 commits into
alibaba:masterfrom
Embedded-AI-Systems:agent
Open

[LLM:Feature] Add Agentic Flow to MNN LLM Chat#4512
huangzhengxiang wants to merge 6 commits into
alibaba:masterfrom
Embedded-AI-Systems:agent

Conversation

@huangzhengxiang

Copy link
Copy Markdown
Contributor

Description

This PR adds an optional Agent mode to MNN LLM Chat while preserving the existing normal chat flow. Agent mode can use host-provided tools for web search,built-in browser reading, Python execution, workspace files, memory, and skill updates. Normal sessions remain backward compatible and continue using the original path when Agent mode is not enabled.

Changes

  • Added per-session Agent mode selection for new conversations.
  • Persisted session-level Agent metadata with database migration and backward compatibility for existing sessions.
  • Added an agentic loop in ChatPresenter:
    • Parses model-emitted system_calls.
    • Executes tool calls step by step.
    • Feeds observations back into the same conversation.
    • Supports visible progress, continuation, budget limits, and stop control.
  • Added robust output parsing for imperfect model JSON:
    • fenced JSON blocks
    • single tool objects
    • nested system_calls arrays
    • loose python_exec recovery
  • Added built-in tools:
    • web_search
    • browser_url
    • python_exec
    • get_current_time
  • Added lightweight Python runtime integration through Chaquopy.
  • Added workspace-aware Python helpers for file generation, including Excel output.
  • Added generated file detection and attachment handling for chat replies.
  • Added an in-app workspace file browser from chat history.
  • Added memory and skill persistence with filtering/compaction to avoid noisy one-off updates.
  • Updated native session handling to better support prompt cache/KV-cache continuation for Agent turns.
  • Added token/performance display for assistant messages.
  • Added docs for Agent design, built-in browser, built-in Python, memory/skill, development notes, and future iteration.
  • Ignored local Android/CMake build artifacts such as .cxx/ and app/standard/.

Notes

  • Existing non-Agent conversations should continue to use the old chat path.
  • Agent mode is selected when creating a new conversation and then persists with that session.
  • Python file outputs are resolved inside the app workspace and surfaced back into chat when possible.
  • The workspace browser now uses the app's internal file browser instead of relying on system file-manager folder intents.

Testing

  • Manual device testing was used during development for:
    • Agent mode selection
    • web/browser tool calls
    • Python execution
    • Excel file generation
    • generated file return in chat
    • workspace file browsing
    • session restore/history behavior

Module

MNN Chat

Type

  • Feature
  • Bugfix
  • Perf
  • Refact
  • Style
  • Doc
  • Test
  • Chore

Checklist

  • Commit message follows [Module:Type] Description format
  • Code compiles without errors
  • Tested on relevant platform(s)
  • No unrelated format or style changes included

@wangzhaode wangzhaode self-assigned this Jun 11, 2026
@wangzhaode wangzhaode requested a review from Juude June 11, 2026 01:40

@wangzhaode wangzhaode left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

这个 PR 为 Android MNN LLM Chat 添加了 Agent 模式,包括 web search、Python 执行、浏览器读取等工具调用能力。代码完全在 apps/Android/MnnLlmChat/ 目录下,不影响 MNN 核心引擎。

需要关注的点

  1. Chaquopy 依赖:引入了 com.chaquo.python 插件(Python on Android),会显著增加 APK 体积。确认是否只在特定 flavor 中启用,以及是否有无 Python 的降级方案。

  2. 安全性:python_exec 工具在设备上执行模型生成的 Python 代码,存在安全风险。建议添加沙箱限制(禁止网络访问、文件系统访问范围限制等)。

  3. Agent loop 限制:确认是否有最大迭代次数限制防止无限循环(budget limits 在描述中提到了,很好)。

  4. 数据库迁移:新增 Agent 相关字段需要确保向后兼容(老版本数据库升级不崩溃)。

  5. build.gradle 格式:从 build.gradle.kts(Kotlin DSL)变成了 build.gradle(Groovy),这是有意的重构还是冲突?请确认。

  6. C++ JNI 层改动:llm_session.cpp 和 llm_mnn_jni.cpp 有少量改动,需要确认不影响现有 chat 流程。

  7. 文档完善 ✅:docs/ 目录下有完整的设计文档。

建议

  • @Juude 的 review 完成后再决定合入
  • 确认格式检查失败的原因并修复
  • 本 PR 不涉及引擎层改动,主要需要 Android 应用侧的测试验证

@Juude

Juude commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了,可以找个在线裁剪平台裁剪下,然后邮件发我下,我上传到 cdn

@huangzhengxiang

Copy link
Copy Markdown
Contributor Author

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了,可以找个在线裁剪平台裁剪下,然后邮件发我下,我上传到 cdn

发你的gmail邮箱了,压缩完2.8MB

@Juude

Juude commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了,可以找个在线裁剪平台裁剪下,然后邮件发我下,我上传到 cdn

发你的gmail邮箱了,压缩完2.8MB

可以直接用这个 cdn 地址: https://meta.alicdn.com/data/mnn/assets/Agent.gif

Added design notes and an agent demo image to the README.
Updated README to improve image alignment and formatting.
@huangzhengxiang

Copy link
Copy Markdown
Contributor Author

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了,可以找个在线裁剪平台裁剪下,然后邮件发我下,我上传到 cdn

发你的gmail邮箱了,压缩完2.8MB

可以直接用这个 cdn 地址: https://meta.alicdn.com/data/mnn/assets/Agent.gif

已更新,把原有gif删了。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants