Replies: 17 comments
|
❌ Evaluation failed An error occurred: Please check the workflow logs. |
|
This is a strange one. I will investigate it soon, but just to be clear has ha-mcp ever worked for you before or is this a fresh install? Trying to see if one of the latest updates maybe caused the issue, or if this is a fresh install. ALso have you rebooted HA? The nabu casa webhook addon requires a reboot of HA, but if that was the issue then you wouldn't be able to use tools or connect at all till you restarted. I did make some recent changes to the nabu casa webhook addon as well pushing it to 1.0.2 so I wonder if that could be causing an issue. I have the webhook app running right now on my installation and I"m not seeing any similar logs but I'll let it run overnight and see if anything happens. |
|
Tried to reproduce all three patterns against the dev addon (Fork-Dev Pattern A — Pattern B — idle exit at ~6 min: Did not reproduce. Server stayed alive across a ~6 min idle window plus follow-up probes. Pattern C — kill ~32 s after a successful call: Did not reproduce. Probes at T+50, T+110, T+230, then a 5-call burst, then 4 min idle, then another probe — all OK. One thing worth flagging on Pattern A specifically: per Plan: optional in-addon kill-source diagnosticsTo make this debuggable on any install hitting it, we'd like to land an opt-in diagnostic mode. Posting the plan here for feedback before opening the PR. User-facing toggle:
Toggle off (default): no behavior change. The ctypes/sigaction path is not installed. Toggle on:
Tests: unit tests for Value is independent of #1109 — even if the cause turns out to be obvious in retrospect, the addon should self-diagnose kill source on any future occurrence so we don't have to ask reporters to set up |
Follow-up: another 1-hour session, 13 restarts — and one captured SIGTERM with exact supervisor correlationCaptured a fresh 1-hour log window (~09:47–11:14) on the Dev channel Add-on settings during the session:
No option changes and no updates during the hour, so neither option-save Caught one in the act: SIGTERM from the supervisor (PID 0, cross-namespace)For the first time, the This correlates sub-second with the supervisor log: So this particular kill is unambiguously: supervisor sends SIGTERM → A few things worth noting from the diagnostics output:
Important caveat — this clean SIGTERM is the exception, not the ruleThe supervisor log shows What's notable is that for all 12 other restarts in this hour, the — note the absence of Cross-checked with supervisor logs — watchdog is not the causeAfter disabling the watchdog mid-session at ~10:48, the add-on continued Phase A — watchdog enabled (until ~10:48): Phase B — watchdog disabled (after ~10:48): Same exit pattern, no watchdog warning, no auto-restart. This rules out What the supervisor log does not showFor every death in both phases, none of these appear at the
The supervisor learns about each unexplained death ~15 seconds after Confirmed observations from the add-on log itself
Additional data pointThe very first start after the add-on came up logs — note: no chain to an existing SIGTERM handler. Every subsequent Summary
Happy to provide both full logs (add-on + supervisor) if useful — let me |
|
❌ Analysis failed An error occurred: Please check the workflow logs. |
|
Sorry I didn't realize that the issue auto-closed. Reopening now till we figure out what is causing this. I will analyze that info and decide further steps |
|
Beautiful capture, thanks. The trapped kill at 10:58:02 is everything we'd hope for from the SA_SIGINFO path: cross-namespace label rendered correctly, sub-second supervisor correlation, clean SIGTERM via the Docker stop path. That one was your manual restart, expected behavior — confirms the feature itself works end-to-end. The 12 untrapped restarts are the actual #1109. Two datapoints from your existing logs would meaningfully narrow this down — both viewable from the HA UI without CLI: 1. Add-on log → does That line is logged by
2. Settings → System → Logs → Host (in the dropdown) — anything around the death timestamps? This surfaces the host's systemd journal, which is where the kernel's OOM killer leaves a record (the addon's supervisor log doesn't include kernel events). Your VmRSS of 76 MB makes cgroup OOM unlikely, but it's a one-glance ruling-out. Acknowledged separately: You're right about the first-boot install gap — Likely follow-up after these two answers: If the kills are confirmed SIGKILL, we can't catch them at-the-moment by design. But we can extend |
|
Thanks a lot for reopening and for digging into this — really appreciate it. |
|
The automated triage hit an internal error. Your issue will be triaged manually — no action needed on your part. 🤖 Automated triage by Issue Bot |
|
Both datapoints checked — results below. 1.
|
|
The automated triage hit an internal error. Your issue will be triaged manually — no action needed on your part. 🤖 Automated triage by Issue Bot |
|
P.S. I've now disabled both Speech-to-Phrase and Matter Server (the two add-ons that dominated the OOM-kill list) and added an 8 GB swap file as a safety net. Will report back if the unexplained MCP restarts continue under those conditions — that should help separate "host RAM pressure side-effect" from "actual #1109 root cause." |
|
🤖 Issue Triage Bot is analyzing this issue... This may take a minute. I'm:
I'll update this comment when complete. |
|
Ok this is pointing more toward an issue with your HA in general. I don't think that Enhancing the advanced debug logging feature will help, but I do think that installing the Glances addon/integration will help you immensely. I personally use it, it gives a large wealth of information and should be able to help you pinpoint what exactly is going on. I think it only shows stuff in realtime by default but I think that there's a way to set it up so it saves logs too. |
|
Any updates on your side? Did killing speech to phrase fix the issue for you? Just wondering if I can close the issue or if we need to do more on our side. |
|
I'm going to convert this to a discussion, please @ me if you have any more issues or feel free to reopen if you feel there's anymore we can do on our side. Hoping the issue is resolved for you now! |
|
After disabling Speech-to-Phrase and Matter Server (the main global_oom offenders) and adding an 8 GB swap file, the MCP server has been running for two days without a single unexplained restart — across idle stretches and tool-heavy bursts. Neither Pattern B nor C has reappeared. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
The MCP server in the add-on exits the entire Uvicorn process under at least three different conditions. With Supervisor watchdog enabled, this triggers a crashloop. With watchdog disabled, the server stays dead until manually restarted. The Webhook Proxy add-on continues running but logs
MCP server unreachableuntil intervention.Observed three distinct crash patterns in a single 7-hour session — at least two of them appear to be separate bugs.
Environment
enable_skills: true,enable_skills_as_tools: true,enable_tool_search: true,backup_hint: normalmcp_servercore integration is not installed/activeThree observed crash patterns
Pattern A —
ToolErrorexits the entire serverA failing tool call raises
ToolError(e.g.CONNECTION_FAILEDbecause the WebSocket tohttp://supervisor/corecouldn't be established). Instead of returning the error to the MCP client, the exception propagates to Uvicorn and shuts down the application.Stack trace points to:
Pattern B — Server exits while completely idle
This is the most surprising one. The server runs for ~6 minutes with zero requests (no tool calls, no health checks visible in the add-on log), then exits.
No
ToolError, no exception, no incoming request. Something is sending SIGTERM to the process. Since Supervisor watchdog is disabled, this is not the supervisor.Pattern C — Wrapper-script kills server after a successful tool call
A tool call succeeds (HTTP 200), then ~32 s later the server is shut down. The log line ordering is the diagnostic clue:
The
[INFO] MCP server stoppedline uses the bash-wrapper log format (different from Uvicorn'sINFO:). It appears before Uvicorn's shutdown messages, indicating the wrapper script killed the Python process and then logged that fact, after which Uvicorn shut down gracefully on the SIGTERM. This means there's a container-internal supervisor/watchdog killing the server, independent of the Supervisor-level watchdog setting.This is consistent with the Webhook Proxy add-on independently observing the server flapping in/out of reachability:
Crashloop with watchdog enabled
When Supervisor watchdog is on, the above patterns cause an endless restart loop. Supervisor logs (24× in 7.5 hours):
Underlying connectivity issues (likely contributing, possibly orthogonal)
The supervisor log shows the WebSocket from the add-on to HA Core flapping:
…repeating every few seconds. The add-on log also shows:
Whether these are the trigger for the crashes or a separate issue isn't fully clear, but they're frequent enough to be relevant context.
Tools observed to trigger crashes
From this user's logs (n=1, anecdotal):
ha_search_entities— confirmed (matches [BUG] MCP connection crashes persistently when calling ha_search_entities tool #406)Automation config fetch budget exhaustedha_get_addon,ha_manage_addon— fail withCONNECTION_FAILEDwhen WS to supervisor is unreachableTools that worked in the same session without crashing:
ha_get_addon(slug=...)— when WS happened to be upha_get_updates()Expected behavior
ToolErrorraised inside a tool handler should be returned to the MCP client and not terminate the server.marked unhealthy because ...) so users can diagnose.Workaround currently in use
enable_tool_search: trueenabled (does not fully solve the problem)Possibly related
All reactions