fix: exit on HA container death + daily reset before CI check (#1295)

* fix(internal): exit on HA container death + daily reset before CI check

- test_env_manager: poll container status every 30s in --no-interactive
  mode; exit(1) when container stops so systemd Restart=on-failure fires
  and starts a fresh HA instance automatically
- setup-ha-mcp.sh: add hamcp-demo-reset.timer (19:00 UTC daily) to
  restart the service 2h before the check-demo-env CI workflow (21:00
  UTC); explicit UTC in OnCalendar= to avoid timezone ambiguity
- README: document reset timer and update service description

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(internal): catch watchdog Docker/IO errors and exit with code 1

Ensures systemd Restart=on-failure triggers when reload() fails
(e.g. Docker daemon unreachable, container removed by OOM killer).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(internal): also catch docker.errors.DockerException in watchdog

docker.errors.DockerException (base class) and subclasses like
InvalidVersion inherit from Exception, not OSError, so they slipped
past the previous handler and exited 0. Adding it explicitly ensures
all Docker SDK errors trigger a systemd restart.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: julienld

tests/lab-setup/README.md

@@ -37,7 +37,7 @@ The setup script is **idempotent** (safe to re-run) and performs:
 3. **Docker** — Via official get.docker.com script
 4. **uv** — Python package manager for running ha-mcp
 5. **ha-mcp repo** — Clones to `~/ha-mcp` (or pulls if it already exists)
-6. **Systemd service** — Creates `hamcp-demo.service` (starts on boot, restarts on failure) and `hamcp-demo-update.timer` (daily at 3am: git pull, docker image prune, service restart)
+6. **Systemd service** — Creates `hamcp-demo.service` (starts on boot, exits+restarts if HA container dies) and two timers: `hamcp-demo-reset.timer` (daily at 19:00 UTC: fresh HA restart, 2h before nightly CI check) and `hamcp-demo-update.timer` (daily at 03:00 UTC: git pull, docker image prune, service restart)
 7. **Caddy** — Reverse proxy with automatic Let's Encrypt TLS for your domain
 8. **Unattended upgrades** — Auto-updates OS packages, reboots at 4am if needed
 9. **Container cleanup** — Removes stale HA containers and any leaked processes
@@ -65,9 +65,13 @@ sudo systemctl restart hamcp-demo
 # Live logs
 sudo journalctl -u hamcp-demo -f
 
-# Weekly update timer — next run and last result
+# Daily update timer (03:00 UTC) — next run and last result
 sudo systemctl list-timers hamcp-demo-update.timer
 sudo journalctl -u hamcp-demo-update --no-pager -n 20
+
+# Daily reset timer (19:00 UTC) — next run and last result
+sudo systemctl list-timers hamcp-demo-reset.timer
+sudo journalctl -u hamcp-demo-reset --no-pager -n 10
 ```
 
 ## Logs

tests/lab-setup/setup-ha-mcp.sh

@@ -129,24 +129,51 @@ SVCEOF
 
 cat > /etc/systemd/system/hamcp-demo-update.timer << SVCEOF
 [Unit]
-Description=HA-MCP Demo Weekly Update Timer
+Description=HA-MCP Demo Daily Update Timer
 
 [Timer]
-OnCalendar=*-*-* 03:00:00
+OnCalendar=*-*-* 03:00:00 UTC
 AccuracySec=1h
 Persistent=true
 
 [Install]
 WantedBy=timers.target
 SVCEOF
 
+# Daily reset at 19:00 UTC — 2 hours before the CI check-demo-env workflow (21:00 UTC).
+# Restarts the service so HA comes up fresh for the nightly health check.
+cat > /etc/systemd/system/hamcp-demo-reset.service << SVCEOF
+[Unit]
+Description=HA-MCP Demo Daily Reset
+After=hamcp-demo.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/systemctl restart hamcp-demo
+SVCEOF
+
+cat > /etc/systemd/system/hamcp-demo-reset.timer << SVCEOF
+[Unit]
+Description=HA-MCP Demo Daily Reset Timer
+
+[Timer]
+OnCalendar=*-*-* 19:00:00 UTC
+AccuracySec=5m
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+SVCEOF
+
 # Remove sudoers rule if it exists from a previous install (no longer needed)
 rm -f /etc/sudoers.d/hamcp-demo
 
 systemctl daemon-reload
 systemctl enable hamcp-demo.service
 systemctl enable hamcp-demo-update.timer
 systemctl start hamcp-demo-update.timer
+systemctl enable hamcp-demo-reset.timer
+systemctl start hamcp-demo-reset.timer
 
 #=============================================================================
 # 7. CADDY

tests/test_env_manager.py

@@ -17,6 +17,7 @@
 import time
 from pathlib import Path
 
+import docker.errors
 import requests
 from testcontainers.core.container import DockerContainer
 
@@ -302,13 +303,26 @@ def main():
         env.print_status()
 
         if args.no_interactive:
-            # Non-interactive mode: just wait for interrupt
+            # Non-interactive mode: wait for interrupt, and exit if the container dies
+            # so systemd's Restart=on-failure kicks in with a fresh instance.
             logger.info("🔄 Running in non-interactive mode. Press Ctrl+C to stop.")
             try:
                 while True:
-                    time.sleep(1)
+                    time.sleep(30)
+                    if env.container:
+                        wrapped = env.container.get_wrapped_container()
+                        wrapped.reload()
+                        if wrapped.status not in ("running",):
+                            logger.error(
+                                f"❌ Container stopped unexpectedly "
+                                f"(status: {wrapped.status}), exiting for restart..."
+                            )
+                            sys.exit(1)
             except KeyboardInterrupt:
                 logger.info("\n🛑 Received interrupt signal")
+            except (OSError, RuntimeError, docker.errors.DockerException) as e:
+                logger.error(f"❌ Watchdog failure: {e}, exiting for restart...")
+                sys.exit(1)
         else:
             # Interactive menu loop
             while True: