Replies: 2 comments 5 replies
|
What Ballob invitation are you referring to? Sorry if I'm forgetting or not understanding what you're referring to there. Anyways, the issue with local-first with AI and HA-MCP is that the most capable AI is cloud based right now. Even the official MCP server that Homeassistant uses as a component (which is very limited compared to ours) is set up so it can connect to an external AI. However, it's entirely possible to keep to local only with HA-MCP, it doesn't really need internet access other than downloading updates or with the HACS stuff. If you truly want to do local only there are many local AI models you can run on a PC but you need a fairly strong GPU to get any useful usage out of it, and they're a lot more susceptible to hallucinations and are extremely finnicky to set up. Just google openwebui and Ollama and go down the rabbit hole. I personally have Ollama with Qwen 3 VL running on my old PC with a RTX 2060 and I use it for some image recognition automations with HA (for example, it will confirm if a detection from my Frigate setup is a real person or not) and I use it for my local voice assistant as well. It works decently well but can take like 30-45 seconds to get any type of response, and sometimes longer when it's interpreting images.. I managed to successfully get it to connect to ha-mcp one day but it had a lot of trouble doing anything because of its tiny context window and it would constantly lose track of what it was doing, so I just stick with using Claude. If I had more RAM ( I think I have 32gb on that PC) it'd probably work better and of course a better GPU would help a lot, but nothing comes close to a cloud based LLM in my personal opinion. Our setup guide does have a section here for OpenWebUI that should help guide you too https://homeassistant-ai.github.io/ha-mcp/setup/ just click OpenWebUI and go thru it once you get a local LLM up and running. |
|
Thanks for your thoughtful reply. I apologize for not making a link to the thread with Paulus. I know this s a long-ish reply but I guess it's a big topic... At this stage of development it's really not surprising the HA-MCP developers rely on using the best models so you don't waste time with the kinds of problems you saw trying things out on your local GPU. Your work is groundbreaking and deserves strong support. In my own case I've chosen to limit myself to using models that I can run locally while I climb the learning curve. I can be accused of excessive caution but perhaps I've learned a bit too much about the security issues that come with breakthrough technologies. My own iron is limited right now (Quadro P2200 and RTX4060) but I'm preparing to upgrade to an RTX6000 Pro or even the deskside successor to the NVIDIA Spark in development. We'll have to see:-) I do run Ollama and OpenWebUI and experiment with different models to understand the lay of the land. My acid document recognition test, actually, is parsing and itemizing grocery receipts. That seems to set a high bar nobody has yet cleared:-) I'll try out the setup guide you pointed me to. To help others like me that have long experience with HA but are climbing the AI/RAG/MCP/Agent learning curve, a short paragraph on the first page of the HA-MCP readme.md to address local-first operation and limitations would be very helpful to set expectations. My question on "How would I know I'm ready to use HA-MCP fully self hosted?" goes deeper than how much GPU memory I have, TOPS ratings, or what parameters (e.g. context depth) I need to configure. It's really an article of faith in HA that when installing and configuring a new integration, app, or even HACS stuff, that the default settings create a more or less locked down state. The user generally has to take action to expose HA to the internet or cause HA to access resources beyond the local network. Some things (e.g. a weather sensor) might need to access an external URL, but that's almost self evident when choosing to add a weather forecast sensor. The attack or exposure surface inherent in even very basic Agentic AI for HA however seems to be immensely larger. Creating a by-default locked down configuration that is useful seems really challenging. It may be way too early to try and answer the "How will I know?" question. I'm no expert but it seems to me there are an awful lot of weak links. Once filesystem and browser access is granted, or script and automation creation/modification is enabled, or even adding Lovelace cards, it seems like all bets are off. For example, I've read some stuff in the HA-MCP discussions about how to keep, e.g. stuff in secrets.txt file, from the models. It was pointed out that when accessing stuff via the HA API stuff that is assumed to be private is redacted. But with file system access this can be easily circumvented. Lovelace cards that inherently need to query external URLs can easily exfiltrate information. Ditto when a browser is enabled as a tool. Perhaps the solution is careful allow-listing in my firewall/IPS. But even this would require Layer 7 understanding so automated exfiltration attempts from HA can be blocked but stuff I do myself from the keyboard or a script I've written is allowed. The mind boggles. The answer might be that it will be a long time, if ever, before the kind of locked down by default configuration most HA functions provide is possible with any useful MCP functionality. Even if that's the case I would support and encourage what you guys are trying to do with HA-MCP. If some useful subset can be discovered it will only be by climbing the learning curve you're on. |
|
Y'know I'm kind of embarrassed that I never caught that thread! That was in my early days here, I'm pretty slow to notice stuff in the discussion section, I mainly focus on issues sections. Another thing I forgot to mention, our toolsearch option is meant to make the context window smaller and make this work better with smaller models...I never personally tried it on my local LLM but I believe @sergeykad has been using it on local LLMs. And actually we have some pretty robust security features accessible in the web UI right now on the stable version. You can enable and disable tools as you please, except for a few mandatory tools. We have tool security policies that can be set up which require manual human approval any time they're triggered, and the policies can be extremely fine grained. When using regular LLMs they usually have this functionality on their side too, but our functionality is purely within ha-mcp. We have an auto backup feature that backs up every automation or script or whatever on every write action as well. The security policies certainly aren't on by default, but you can see them up before using anything and before exposing your URL anywhere...just simply installing the addon doesn't really open up any risk, you need an LLM to talk to it before it can do anything. I encourage you to install the addon and open the web UI and dig through the various settings, you can definitely use that to completely lock it down. Or do the pypi/docker version and look at the web UI thru there. I also encourage you to please file issues to request some things that you think may be helpful, we review and address every issue, even the all too common ones that are basically troubleshooting things outside of our scope (I. E. Helping people learn to unlock USA IP addresses from cloudflare). |
|
I feel encouraged to experiment a bit more:-) If the manual review policy is the default that would be helpful. What I may do is spin up a separate Proxmox VM just to try things out. That way I can lock everything down pretty easily (in theory:-)). To be fair, I may not get to try it out for a while. The rapid back and forth to my OP is really appreciated but I'm doing a pretty extensive home renovation so there's plenty to distract me. Without this conversation however I probably wouldn't have attempted any experimentation. |
|
External data flowing into HA (calendar events, third-party notifications) can carry injected instructions that redirect an agent's actions, including exfiltrating your data. In a fully self-hosted setup with no external feeds, the risk is minimal. Before connecting a cloud LLM to your HA data, read the provider's data retention and training-opt-out policy. Some plans exclude your inputs from training by default; others do not. The policy will state exactly what leaves your home and for how long. The RTX Pro 6000 (96 GB VRAM) far exceeds what HA workloads require. An RTX Pro 5000 (48 GB) runs every capable open-weight model with room to spare; for HA-MCP specifically, even a consumer RTX 4090 (24 GB) handles typical context sizes comfortably. |
|
Would this strategy for building a sandbox work for learning and familiarization?
If / when I put together items 1-6 I could export it as an .ova, I guess, for others to use, but one provided by the HA-MCP project would be more likely to be trusted. Besides, folks would have to configure the GPU passthrough themselves because it's so system specific. Is this a reasonable way to set up a "safer" sandbox? It would be interesting to get someone to run Mistral from inside the sandbox and pentest it from outside to see how secure it really is. |
|
If you want military-grade security, that's a good way to do it. However, I'm not sure if Qwen3 will run on a 2019-era GPU. You might need to use Qwen2.5 instead, and Ollama support for such an old GPU is also not guaranteed. Furthermore, 5GB of VRAM is very limiting, so you should not expect much from the LLM models that can fit into it. |
Uh oh!
There was an error while loading. Please reload this page.
HA-MCP is justifiably getting rave reviews. What it's trying to do seems to be dead on-target for what I'd like to be able to do as long as I'm comfortable relaxing Home Assistant's local-first philosophy. The control of security and privacy that HA provides currently is extremely important to me however. In fact, the first thing I looked for when studying HA-MCP was information on how to maintain Home Assistant's local-first character while taking advantage of what HA-MCP offers. I didn't see anything along those lines in the docs. Normally if an integration or add-on needs to break that fourth wall it's either self evident or called out clearly early in the documentation. It wasn't until I saw the response to Ballob's invitation to core-ify HA-MCP that it became clear that local-first wasn't yet tested.
I already run locally any AI model I care about. Cloning any tool, add-on, or best practice MCP repository is perfectly OK for me. However, I'm a NOOB when it comes to Agentic AI and it seems to me that controls over unwanted exfiltration are limited to nonexistent when it comes to using agents in general.
So, to put it simply, what should I look for to be comfortable that adopting some or all of what HA-MCP offers won't break the local-first rule?
All reactions