test(e2e): add behavioral cross-check + collection floor to dispatch smoke

The original smoke test (test_backend_dispatch_matches_workflow_env) only
asserted the ``backend`` field that conftest sets. That guards against
the dispatch picking the wrong code path, but it can't catch the case
where conftest reports backend=X while a different HA instance is
actually running.

Adds two layered guards:

- test_supervisor_addon_tool_behavior_matches_backend: calls
  ha_get_addon (which only works when a real Supervisor is running) and
  asserts success-vs-failure matches the claimed backend.
  - HAOS external + inaddon: tool returns success=True with non-empty
    addons list (the bake installs Node-RED, ESPHome, etc.)
  - testcontainer: tool returns success=False because no Supervisor
    proxy exists (RESOURCE_NOT_FOUND from supervisor/api WebSocket call)

  The asymmetry makes it impossible for one backend to impersonate the
  other while keeping the test green — exactly the "goes both ways"
  guarantee.

- test_session_collected_test_count_above_floor: asserts
  request.session.items length >= 850 (current baseline is ~913 across
  all lanes). Catches collection-time regressions where a test file
  fails to import and pytest silently drops tens of tests, leaving the
  suite green with reduced coverage.

All three tests stay under basic/ (no auto-applied haos_only marker)
so they run unconditionally on every lane.

Author: kingpanther13

tests/src/e2e/basic/test_backend_dispatch_smoke.py

@@ -1,4 +1,4 @@
-"""Verifies the conftest backend dispatch picks the backend CI asked for.
+"""Multi-layer smoke tests for backend dispatch correctness.
 
 The three e2e CI lanes set env vars that ``conftest.ha_container_with_fresh_config``
 reads to choose a backend:
@@ -9,48 +9,60 @@
 | haos-e2e-tests.yml (external) | set                  | unset          | ``haos``         |
 | haos-e2e-inaddon-tests.yml    | set                  | ``inaddon``    | ``haos_inaddon`` |
 
-Without an explicit assertion, the dispatch has silent-failure modes that
-all leave CI green while running tests against the wrong backend:
-
-1. ``HAOS_TEST_IMAGE_PATH`` set but ``is_haos_backend_selected()`` returns
-   False (env-var name drift, import-time bug) — both HAOS lanes silently
-   fall through to the testcontainer path.
-2. ``HAOS_TEST_MODE=inaddon`` set but ``is_haos_inaddon_mode()`` reads a
-   different name — inaddon lane silently runs the external dispatch,
-   so ``mcp_client`` talks to the in-process FastMCP server instead of
-   the addon's HTTP endpoint. The whole inaddon integration is untested.
-3. Inaddon dispatch reached but ``addon_mcp_url`` never populated —
-   downstream fixtures route to the wrong endpoint and surface as
-   confusing errors later.
-
-This file is placed under ``basic/`` (NOT ``haos_only/``) on purpose:
-the auto-applied ``haos_only`` marker in ``conftest.pytest_collection_modifyitems``
-would skip the test whenever ``is_haos_backend_selected()`` returns False,
-which is exactly the silent-failure case (1) above. We want the test to
-RUN on every lane and FAIL when the backend doesn't match the env.
+Three layers of guard, each catching a different silent-failure mode:
+
+1. ``test_backend_dispatch_matches_workflow_env`` — asserts the ``backend``
+   field that conftest sets matches what the workflow env vars imply.
+   Catches dispatch falling through to the wrong code path.
+
+2. ``test_supervisor_addon_tool_behavior_matches_backend`` — calls
+   ``ha_get_addon`` (which only works when a real Supervisor is present)
+   and asserts success-vs-failure matches the claimed backend. Catches
+   the case where conftest reports ``backend=X`` but actually a different
+   HA instance is running (e.g. mock Supervisor on testcontainer making
+   addon calls succeed when the real backend has no Supervisor at all,
+   or HAOS reporting itself as container).
+
+3. ``test_session_collected_test_count_above_floor`` — asserts the
+   session collected at least the baseline number of tests. Catches
+   collection-time regressions where a test file fails to import and
+   pytest silently drops tens of tests.
+
+This file is placed under ``basic/`` (NOT ``haos_only/``) on purpose: the
+auto-applied ``haos_only`` marker would skip these whenever
+``is_haos_backend_selected()`` returns False, which is exactly the
+silent-failure case we want to catch. ``basic/`` has no auto-applied
+markers so the tests run unconditionally on every lane.
 """
 
 from __future__ import annotations
 
 import os
 from typing import Any
 
+from ..utilities.assertions import safe_call_tool
+
+# Floor for total collected tests across all lanes. As of 2026-05-22 each
+# lane collects ~913 tests (just differing skip mix per mode). Set well
+# below current value to allow normal test-add/remove churn while still
+# catching the case where ~50+ tests vanish from collection.
+_COLLECTION_FLOOR = 850
+
 
 def test_backend_dispatch_matches_workflow_env(
     ha_container_with_fresh_config: dict[str, Any],
 ) -> None:
     """Conftest dispatch must pick the backend the workflow env implies.
 
-    Runs unconditionally on every lane — assertion branches off the env
-    vars to mirror conftest's own dispatch logic. Mismatch means the
-    dispatch silently picked a different backend than CI asked for.
+    Runs unconditionally on every lane — branches off env vars to mirror
+    conftest's own dispatch logic. Mismatch means the dispatch silently
+    picked a different backend than CI asked for.
     """
     image_path = os.environ.get("HAOS_TEST_IMAGE_PATH")
     mode = os.environ.get("HAOS_TEST_MODE", "")
     backend = ha_container_with_fresh_config["backend"]
 
     if image_path and mode == "inaddon":
-        # haos-e2e-inaddon-tests.yml lane
         assert backend == "haos_inaddon", (
             f"Workflow set HAOS_TEST_IMAGE_PATH + HAOS_TEST_MODE=inaddon "
             f"but dispatch picked backend={backend!r}. The inaddon "
@@ -64,24 +76,81 @@ def test_backend_dispatch_matches_workflow_env(
         )
         assert ha_container_with_fresh_config["container"] is None
     elif image_path:
-        # haos-e2e-tests.yml (external) lane
         assert backend == "haos", (
             f"Workflow set HAOS_TEST_IMAGE_PATH but dispatch picked "
             f"backend={backend!r}. The lane silently fell through to "
             f"the testcontainer path; tests are running against the "
             f"wrong HA instance."
         )
-        # External HAOS sets the testcontainer keys to None.
         assert ha_container_with_fresh_config["container"] is None
         assert ha_container_with_fresh_config["port"] is None
         assert ha_container_with_fresh_config["config_path"] is None
-        # addon_mcp_url is the inaddon-only routing key.
         assert ha_container_with_fresh_config["addon_mcp_url"] is None
     else:
-        # e2e-tests.yml (testcontainer) lane
         assert backend == "container", (
             f"No HAOS env vars set, expected testcontainer backend, "
             f"got backend={backend!r}."
         )
         assert ha_container_with_fresh_config["container"] is not None
         assert ha_container_with_fresh_config["port"] is not None
+
+
+async def test_supervisor_addon_tool_behavior_matches_backend(
+    mcp_client: Any,
+    ha_container_with_fresh_config: dict[str, Any],
+) -> None:
+    """Behavioral cross-check: ``ha_get_addon`` must succeed on HAOS, fail on container.
+
+    Stronger guarantee than the dispatch-field check: conftest could
+    self-report ``backend=container`` but actually have HAOS running
+    (or vice versa). A real Supervisor only exists on HAOS — the HA
+    Core testcontainer has no Supervisor service running. So:
+
+    - HAOS external + inaddon: ``ha_get_addon`` returns a populated
+      addons list (the bake installs several addons).
+    - testcontainer: ``ha_get_addon`` returns ``success=False`` because
+      the Supervisor proxy endpoint is unreachable.
+
+    The asymmetry of this check makes it impossible for one backend to
+    impersonate the other while keeping this test green.
+    """
+    backend = ha_container_with_fresh_config["backend"]
+    result = await safe_call_tool(mcp_client, "ha_get_addon", {})
+
+    if backend in ("haos", "haos_inaddon"):
+        assert result.get("success") is True, (
+            f"ha_get_addon failed on {backend} backend; Supervisor must "
+            f"be running. Result: {result!r}"
+        )
+        # list_addons returns ``{"success": True, "addons": [...], "summary": {...}}``
+        # — ``addons`` is a top-level key, not nested under ``data``.
+        addons = result.get("addons") or []
+        assert isinstance(addons, list) and len(addons) > 0, (
+            f"Expected installed addons on {backend} (the bake installs "
+            f"Node-RED, ESPHome, AppDaemon, dev addon, etc.); got "
+            f"{addons!r}"
+        )
+    else:
+        # testcontainer has no Supervisor → ha_get_addon must fail
+        assert result.get("success") is False, (
+            f"ha_get_addon unexpectedly succeeded on {backend} backend. "
+            f"Testcontainer has no Supervisor service; success here "
+            f"means we're actually running on HAOS but conftest reported "
+            f"backend={backend!r}. Result: {result!r}"
+        )
+
+
+def test_session_collected_test_count_above_floor(request: Any) -> None:
+    """Session collected at least ``_COLLECTION_FLOOR`` tests.
+
+    Catches collection-time regressions: a test file fails to import,
+    pytest collects tens of fewer tests, the suite stays green with
+    reduced coverage. Collection count is mode-independent (all lanes
+    collect the same items, mode only changes pass/skip mix).
+    """
+    total = len(request.session.items)
+    assert total >= _COLLECTION_FLOOR, (
+        f"Only {total} tests collected, expected >= {_COLLECTION_FLOOR}. "
+        f"A test file likely failed to import, dropping coverage. Check "
+        f"for collection errors in the pytest output."
+    )