Skip to content

Add embedded MCP server for in-process UI introspection#11217

Open
tilladam wants to merge 8 commits intoslint-ui:masterfrom
tilladam:mcp-server-simplify
Open

Add embedded MCP server for in-process UI introspection#11217
tilladam wants to merge 8 commits intoslint-ui:masterfrom
tilladam:mcp-server-simplify

Conversation

@tilladam
Copy link
Copy Markdown
Contributor

Summary

  • Add an in-process MCP (Model Context Protocol) server to the Slint testing backend, allowing MCP clients (e.g. Claude Code, GitHub Copilot) to inspect and interact with a running Slint application's UI directly
  • When the mcp feature is enabled and SLINT_MCP_PORT=<port> is set at runtime, an HTTP server starts on 127.0.0.1:<port> implementing MCP Streamable HTTP transport with 11 tools
  • Refactors systest to share an IntrospectionState with the MCP transport, eliminating duplicated window/element tracking logic

Architecture

selector/mcp → testing/mcp
                 ├── introspection.rs  (shared state: arenas, properties, queries, conversions)
                 ├── mcp_server.rs     (HTTP server, JSON-RPC 2.0, 11 MCP tools)
                 └── systest.rs        (refactored to delegate to shared IntrospectionState)

MCP tools

Tool Description
list_windows Discover open windows
get_window_properties Size, position, root element handle
get_element_tree Flat list of UI hierarchy with properties
get_element_properties Full details on a single element
find_elements_by_id Look up elements by qualified ID
query_element_descendants Pipeline-based search by type, ID, or role
take_screenshot PNG screenshot as inline MCP image
click_element Simulate mouse click
invoke_accessibility_action Default, Increment, Decrement, Expand
set_element_value Set text input content, slider value, etc.
dispatch_key_event Send keyboard events

Key design decisions

  • Shared introspection layer: element_properties(), query_element_descendants(), role/button/action conversions all live in introspection.rs, called by both transports.
  • Localhost-only: Binds to 127.0.0.1, validates Origin header for DNS rebinding protection.
  • Arena eviction: FIFO eviction with 10k handle cap; root handles preserved.
  • Rich MCP instructions: The initialize response includes detailed workflow guidance, handle format, enum values, query syntax, and interaction tips — consumed by all MCP clients.

Test plan

  • cargo build -p i-slint-backend-testing --features system-testing — systest compiles
  • cargo build -p i-slint-backend-testing --features mcp — MCP compiles
  • cargo test -p i-slint-backend-testing --features system-testing,mcp — 19 tests pass
  • Manual integration test against gallery example: list windows, get tree, query 87 buttons, click, set text input value, take screenshots, dispatch key events — all working

tilladam and others added 4 commits April 10, 2026 12:18
Add an embedded MCP (Model Context Protocol) server that allows
MCP clients (e.g. Claude Code) to inspect and interact with running
Slint applications via HTTP/JSON-RPC.

The MCP server uses the pbjson serde impls from PR 2 to serialize
proto types directly as JSON — no hand-written parameter structs or
manual JSON conversion. Both the systest (binary protobuf over TCP)
and MCP (JSON over HTTP) transports share a common introspection
layer for window/element tracking with FIFO eviction.

New files:
- introspection.rs: shared IntrospectionState with arena management
- mcp_server.rs: thin JSON-RPC/HTTP wrapper with 11 MCP tools

Enable with: SLINT_MCP_PORT=8080 ./your-slint-app
@tilladam tilladam force-pushed the mcp-server-simplify branch from bcccaf1 to 5413ba0 Compare April 10, 2026 10:26
tilladam and others added 4 commits April 10, 2026 12:51
Move window_properties() and take_snapshot_response() into
IntrospectionState so both transports call the same method instead of
duplicating the logic. Remove redundant index_to_handle/handle_to_index
wrappers and physical size/position helpers — import directly from
introspection. Move accessibility role mapping test to introspection.rs
where the function under test lives.
@tronical
Copy link
Copy Markdown
Member

Thanks, is is I think as reduced as it gets without "Handarbeit" :). I think we should take this in and start experimenting and learning :)

Copy link
Copy Markdown
Member

@tronical tronical left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Merging contingent on clippy being green, the servo failure is unrelated)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants