Skip to content

Commit 39417ff

Browse files
julienldclaude
andcommitted
test: fix three categories of E2E test flakiness
**Category A — external network (7 incidents in 40 days):** test_import_blueprint_saves_to_disk fetched a live GitHub gist that was occasionally unreachable from CI runners. Replace with a locally-served blueprint file (tests/assets/blueprints/e2e_test_blueprint.yaml) via an HTTP server started in _blueprint_http_server. The server host IP detection handles two Docker environments: - Docker Desktop (WSL2/Mac): host.docker.internal resolves via Docker's embedded DNS — probe a container and use the hostname directly without overriding it via extra_hosts (doing so breaks Docker Desktop routing) - Plain Linux Docker (CI): inject host.docker.internal via extra_hosts so the mapping exists inside the HA container **Category B — ha_mcp_tools startup race (4+ incidents, worse on ARM):** conftest installed the custom component but did not wait for HA to register its services. Tests that call ha_config_set_yaml immediately after container start got COMPONENT_NOT_INSTALLED errors. Add a 30s poll for ha_mcp_tools domain in /api/services before yielding the fixture. **Category C — HA entity/service startup race (3 incidents):** - sun.sun remained in 'unknown' state while template tests asserted above_horizon/below_horizon — add a 30s poll until sun.sun leaves 'unknown' - input_boolean services sometimes not ready when helpers bulk tests ran — add a 30s poll for required service domains (input_boolean, sun) All three waits are soft: they log a warning on timeout rather than failing the fixture, preserving existing test failure semantics. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 77abe0b commit 39417ff

3 files changed

Lines changed: 216 additions & 7 deletions

File tree

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
blueprint:
2+
name: E2E Test Blueprint
3+
description: Minimal automation blueprint used by ha-mcp E2E tests. Safe to delete.
4+
domain: automation
5+
input:
6+
target_entity:
7+
name: Target Entity
8+
description: The entity to control
9+
selector:
10+
entity: {}
11+
12+
trigger:
13+
- platform: time
14+
at: "00:00:00"
15+
16+
action:
17+
- service: homeassistant.turn_on
18+
target:
19+
entity_id: !input target_entity

tests/src/e2e/conftest.py

Lines changed: 191 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,18 @@
1717
"""
1818

1919
import asyncio
20+
import http.server
2021
import json
2122
import logging
2223
import os
2324
import shutil
25+
import socket
2426
import sys
2527
import tempfile
28+
import threading
2629
import time
2730
from collections.abc import AsyncGenerator
31+
from functools import partial
2832
from pathlib import Path
2933
from typing import Any
3034

@@ -184,8 +188,84 @@ async def test_settings():
184188
return settings
185189

186190

191+
def _detect_docker_host() -> dict:
192+
"""Detect the correct host address and extra_hosts config for the Docker environment.
193+
194+
Docker Desktop (WSL2 / Mac / Windows) embeds a DNS server that resolves
195+
``host.docker.internal`` inside containers automatically. On plain Linux
196+
Docker (GitHub Actions CI) that DNS is absent, so we must inject the
197+
mapping via ``--add-host host.docker.internal:host-gateway``.
198+
199+
Strategy: run a minimal probe container and ask it to resolve
200+
``host.docker.internal``. If it resolves, Docker Desktop DNS is active and
201+
we must NOT override the entry (doing so breaks the internal routing). If
202+
it does not resolve, we are on plain Linux Docker and must add extra_hosts.
203+
204+
Returns a dict with:
205+
- ``hostname`` – hostname that Docker containers use to reach the host
206+
- ``extra_hosts`` – dict passed to ``container.with_kwargs`` (may be empty)
207+
"""
208+
import subprocess
209+
210+
try:
211+
result = subprocess.run(
212+
[
213+
"docker", "run", "--rm", "alpine",
214+
"sh", "-c",
215+
"getent hosts host.docker.internal 2>/dev/null | awk '{print $1}'",
216+
],
217+
capture_output=True,
218+
text=True,
219+
timeout=30,
220+
)
221+
if result.stdout.strip():
222+
# Docker Desktop DNS resolved the name — use hostname, no override needed
223+
logger.info("🔍 Docker Desktop DNS detected — using host.docker.internal as-is")
224+
return {"hostname": "host.docker.internal", "extra_hosts": {}}
225+
except Exception as exc:
226+
logger.debug(f"Docker Desktop DNS probe failed: {exc}")
227+
228+
# Plain Linux Docker — inject the mapping so the hostname resolves in the container
229+
logger.info("🔍 Plain Linux Docker detected — injecting host.docker.internal via extra_hosts")
230+
return {
231+
"hostname": "host.docker.internal",
232+
"extra_hosts": {"host.docker.internal": "host-gateway"},
233+
}
234+
235+
187236
@pytest.fixture(scope="session")
188-
def ha_container_with_fresh_config():
237+
def _blueprint_http_server():
238+
"""Start a local HTTP server for blueprint files before the HA container launches.
239+
240+
Must start before the container so the port is known when ``extra_hosts``
241+
is configured in ``ha_container_with_fresh_config``.
242+
"""
243+
env = _detect_docker_host()
244+
245+
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
246+
s.bind(("", 0))
247+
port = s.getsockname()[1]
248+
249+
assets_dir = Path(__file__).parent.parent.parent / "assets" / "blueprints"
250+
assets_dir.mkdir(parents=True, exist_ok=True)
251+
252+
handler = partial(http.server.SimpleHTTPRequestHandler, directory=str(assets_dir))
253+
handler.log_message = lambda *args: None # type: ignore[method-assign]
254+
srv = http.server.HTTPServer(("0.0.0.0", port), handler)
255+
t = threading.Thread(target=srv.serve_forever, daemon=True)
256+
t.start()
257+
258+
base_url = f"http://{env['hostname']}:{port}"
259+
logger.info(f"🌐 Blueprint HTTP server on :{port}, container URL: {base_url}")
260+
261+
try:
262+
yield {"base_url": base_url, "port": port, "extra_hosts": env["extra_hosts"]}
263+
finally:
264+
srv.shutdown()
265+
266+
267+
@pytest.fixture(scope="session")
268+
def ha_container_with_fresh_config(_blueprint_http_server):
189269
"""Create Home Assistant container with fresh config using testcontainers."""
190270
# --- Safety guard 1: ensure Docker is available before doing anything else ---
191271
try:
@@ -280,8 +360,15 @@ def ha_container_with_fresh_config():
280360
str(config_path), "/config", "rw"
281361
) # Ensure read-write mount
282362
container = container.with_env("TZ", "UTC")
283-
# Add privileged mode for Home Assistant hardware access
284-
container = container.with_kwargs(privileged=True)
363+
# Add privileged mode for Home Assistant hardware access.
364+
# On plain Linux Docker (CI) also inject the host.docker.internal mapping so
365+
# the blueprint HTTP server is reachable from within the container.
366+
# On Docker Desktop the mapping is provided by Docker's embedded DNS and must
367+
# NOT be overridden here.
368+
container_kwargs: dict = {"privileged": True}
369+
if _blueprint_http_server.get("extra_hosts"):
370+
container_kwargs["extra_hosts"] = _blueprint_http_server["extra_hosts"]
371+
container = container.with_kwargs(**container_kwargs)
285372

286373
# Remove any .HA_RESTORE file that might cause issues
287374
restore_file = config_path / ".HA_RESTORE"
@@ -444,12 +531,100 @@ def ha_container_with_fresh_config():
444531
f"(minimum: {MIN_ENTITIES}). Check Docker logs."
445532
)
446533

534+
# Wait for key HA service domains to register. Components loaded and
535+
# entities present does not guarantee services are ready — individual
536+
# integrations (input_boolean, sun) register their services
537+
# asynchronously after their entities appear.
538+
REQUIRED_SERVICES = {"input_boolean", "sun"}
539+
SERVICE_WAIT = 30
540+
logger.info("⏳ Waiting for required service domains to register...")
541+
for svc_attempt in range(SERVICE_WAIT):
542+
try:
543+
svc_resp = requests.get(
544+
f"{base_url}/api/services", timeout=5, headers=headers
545+
)
546+
if svc_resp.status_code == 200:
547+
registered = {s.get("domain") for s in svc_resp.json()}
548+
missing = REQUIRED_SERVICES - registered
549+
if not missing:
550+
logger.info(
551+
f"✅ Required service domains ready after {svc_attempt + 1}s"
552+
)
553+
break
554+
if svc_attempt % 5 == 0:
555+
logger.info(
556+
f"⏳ Waiting for service domains: {missing}"
557+
)
558+
except (requests.exceptions.RequestException, json.JSONDecodeError) as exc:
559+
logger.debug(f"Service check failed: {exc}")
560+
time.sleep(1)
561+
else:
562+
logger.warning(
563+
f"⚠️ Service domain wait timed out after {SERVICE_WAIT}s "
564+
f"— some tests may be flaky"
565+
)
566+
567+
# Wait for ha_mcp_tools custom component services (installed above).
568+
# The component is loaded after core services, so it needs its own check.
569+
HA_MCP_TOOLS_WAIT = 30
570+
ha_mcp_tools_src = repo_root / "custom_components" / "ha_mcp_tools"
571+
if ha_mcp_tools_src.exists():
572+
logger.info("⏳ Waiting for ha_mcp_tools services to register...")
573+
for mcp_attempt in range(HA_MCP_TOOLS_WAIT):
574+
try:
575+
svc_resp = requests.get(
576+
f"{base_url}/api/services", timeout=5, headers=headers
577+
)
578+
if svc_resp.status_code == 200:
579+
domains = {s.get("domain") for s in svc_resp.json()}
580+
if "ha_mcp_tools" in domains:
581+
logger.info(
582+
f"✅ ha_mcp_tools services ready after {mcp_attempt + 1}s"
583+
)
584+
break
585+
except (requests.exceptions.RequestException, json.JSONDecodeError) as exc:
586+
logger.debug(f"ha_mcp_tools service check failed: {exc}")
587+
time.sleep(1)
588+
else:
589+
logger.warning(
590+
f"⚠️ ha_mcp_tools services not registered after {HA_MCP_TOOLS_WAIT}s "
591+
f"— yaml config tests may fail"
592+
)
593+
594+
# Wait for sun.sun to leave the 'unknown' state. During HA startup the
595+
# sun integration reports 'unknown' until it computes the first position.
596+
# Template tests that assert above/below_horizon will fail if we proceed
597+
# before the sun integration finishes its first calculation.
598+
SUN_WAIT = 30
599+
logger.info("⏳ Waiting for sun.sun to reach a known state...")
600+
for sun_attempt in range(SUN_WAIT):
601+
try:
602+
sun_resp = requests.get(
603+
f"{base_url}/api/states/sun.sun", timeout=5, headers=headers
604+
)
605+
if sun_resp.status_code == 200:
606+
sun_state = sun_resp.json().get("state", "unknown")
607+
if sun_state != "unknown":
608+
logger.info(
609+
f"✅ sun.sun is '{sun_state}' after {sun_attempt + 1}s"
610+
)
611+
break
612+
except (requests.exceptions.RequestException, json.JSONDecodeError) as exc:
613+
logger.debug(f"sun.sun check failed: {exc}")
614+
time.sleep(1)
615+
else:
616+
logger.warning(
617+
f"⚠️ sun.sun still 'unknown' after {SUN_WAIT}s "
618+
f"— template tests may fail"
619+
)
620+
447621
# Store connection info for other fixtures
448622
container_info = {
449623
"container": container,
450624
"port": host_port,
451625
"base_url": base_url,
452626
"config_path": str(config_path),
627+
"blueprint_server": _blueprint_http_server,
453628
}
454629

455630
try:
@@ -759,3 +934,16 @@ async def _wait_for_state(
759934
return False
760935

761936
return _wait_for_state
937+
938+
939+
@pytest.fixture(scope="session")
940+
def local_blueprint_server(ha_container_with_fresh_config):
941+
"""Return blueprint HTTP server info for tests that need to import blueprints.
942+
943+
The server is started by ``_blueprint_http_server`` before the HA container
944+
and stored in ``ha_container_with_fresh_config``; this fixture simply exposes
945+
it so tests don't need to depend on ``ha_container_with_fresh_config`` directly.
946+
"""
947+
server = ha_container_with_fresh_config["blueprint_server"]
948+
logger.info(f"🌐 Blueprint server at {server['base_url']}")
949+
yield server

tests/src/e2e/workflows/blueprints/test_blueprints.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -247,18 +247,20 @@ async def test_import_blueprint_nonexistent_url(self, mcp_client):
247247
logger.info("ha_import_blueprint properly handles non-existent URL")
248248

249249
@pytest.mark.slow
250-
async def test_import_blueprint_saves_to_disk(self, mcp_client):
250+
async def test_import_blueprint_saves_to_disk(self, mcp_client, local_blueprint_server):
251251
"""
252252
Test: Import blueprint actually saves to disk (issue #685)
253253
254254
Validates that ha_import_blueprint calls both blueprint/import (validate)
255255
AND blueprint/save (persist), so the blueprint appears in the list.
256-
Uses a known community blueprint that won't already be installed.
256+
Uses a locally-served blueprint file to avoid external network dependencies.
257257
"""
258258
logger.info("Testing ha_import_blueprint saves blueprint to disk...")
259259

260-
# Use a community blueprint unlikely to be pre-installed
261-
test_url = "https://gist.github.com/Blackshome/4010fb83bb8c19b5fa1425526c6ff0e2"
260+
# Serve the blueprint from a local HTTP server accessible by the HA container.
261+
# This avoids flaky failures caused by transient GitHub network issues on CI.
262+
test_url = f"{local_blueprint_server['base_url']}/e2e_test_blueprint.yaml"
263+
logger.info(f"Using local blueprint URL: {test_url}")
262264

263265
async with MCPAssertions(mcp_client) as mcp:
264266
# List blueprints before import

0 commit comments

Comments
 (0)