Skip to content

Commit 6f9ca4f

Browse files
authored
test(e2e): add A2 negative-input tests for optional-id list-or-detail tools (#1058)
* test(e2e): add A2 negative-input tests for optional-id list-or-detail tools Closes the A2 (optional ID → list or detail) negative-input test gaps from #914. Two existing soft tests are hardened with explicit error-code and message-substring assertions; three new tests are added for tools that had no negative-input coverage on the single-ID lookup branch. - Hardened test_get_blueprint_not_found in blueprints/test_blueprints.py — added error.code == 'RESOURCE_NOT_FOUND' assertion alongside the existing suggestions check. - Hardened test_get_nonexistent_label in config/test_label_crud.py — added error.code == 'ENTITY_NOT_FOUND' assertion plus message-substring check on top of the existing success check. - New TestGetIntegrationNegativeInputs in integrations/test_list_integrations.py with test_get_integration_nonexistent_entry_id. The existing tests in this file exercise list/query/domain-filter modes (A4-style empty results), not the entry_id single-entry lookup branch, which raises RESOURCE_NOT_FOUND via _get_single_entry's 404 catch. - New TestDeviceGetNegativeInputs in registry/test_device_registry.py with test_get_device_nonexistent_device_id. Single-device lookup raises ENTITY_NOT_FOUND when the device_id is absent from the registry list. - New TestUpdatesGetNegativeInputs in updates/test_updates.py with test_get_updates_nonexistent_entity_id. _get_update_details fetches the entity state via REST; a 404 response (HomeAssistantAPIError) is mapped to ENTITY_NOT_FOUND in the except-Exception branch. Live-probed against a real HA instance: GET /api/states/update.<nonexistent> returns HTTP 404 with body {"message": "Entity not found."}. ha_get_zone is intentionally not included — its A2 negative path is already hard-asserted in zones/test_lifecycle.py. ha_config_get_category, ha_get_todo, ha_get_entity_exposure and ha_config_get_dashboard were deliberately scoped out (already-hard delete-side coverage in different file, validation-pathway distinct from A2-classic, legitimate default-exposure return shape, and a separate maintainer-mandated PR for dashboard_id dual-accept respectively). References #914. * test(e2e): address Gemini review on A2 tests — structured error access Addresses 5 medium-priority Gemini Code Assist findings on PR #1058: - blueprints: drop redundant 'not found' substring assertion (already validated by call_tool_failure(expected_error=...)); switch to direct result['error'] access instead of .get() since the helper guarantees the key. - label_crud, list_integrations, device_registry, updates: replace str(data.get('error','')).lower() with data['error']['message'].lower() for structured field access. Repo precedent: automation/test_lifecycle.py uses the same pattern. No behavioral change — same assertions, more robust access path. * test(e2e): address Gemini follow-up — assert structured suggestion presence Addresses 4 medium-priority Gemini Code Assist findings on PR #1058 re-review: - label_crud, list_integrations, device_registry: add 'suggestion' or 'suggestions' presence assertion to verify the styleguide.md progressive disclosure principle (error responses guide next steps). - updates: switch from safe_call_tool to MCPAssertions.call_tool_failure for consistency with the file's existing test style; add the same suggestion presence assertion. Note on the assertion shape: create_error_response (errors.py:243-248) sets the singular 'suggestion' key for any non-empty suggestions list and the plural 'suggestions' key only when len > 1. The four targeted tools each provide exactly one suggestion in their not-found path, so a strict 'suggestions' (plural) assertion would fail; the disjunction matches both forms and is robust against future single->multi migrations. * test(e2e): align blueprint suggestion assertion with disjunction pattern Addresses Gemini follow-up finding on test_blueprints.py:188. The plural-only 'suggestions' check is fragile — create_error_response (errors.py:243-248) only sets the plural key when len > 1. ha_get_blueprint currently provides two suggestions in its not-found path (blueprints.py:147) so the plural assertion passes today, but the test would silently break if a future change reduces the count to one. Switch to the same 'suggestion' or 'suggestions' disjunction pattern used in the other four negative-input tests for consistency and robustness. * test(e2e): align docstring with current assertion shape Drift fix: docstring previously claimed 'message-substring assertions' which was removed earlier in this PR (redundant to call_tool_failure's expected_error parameter). Update wording to reflect the actual hardening — error-code assertion plus structured suggestion-presence disjunction (suggestion or suggestions). * test(e2e): simplify suggestion assertions to singular key Addresses 4 medium-priority Gemini findings — the 'suggestion' or 'suggestions' disjunction is redundant. create_error_response (errors.py:243-248) sets the singular 'suggestion' key unconditionally when any suggestions are provided; the plural 'suggestions' key is only set additionally when len > 1. Checking the singular key is both sufficient and aligned with the helper's contract. Applied uniformly across all 5 tests for consistency.
1 parent 0950652 commit 6f9ca4f

5 files changed

Lines changed: 173 additions & 8 deletions

File tree

tests/src/e2e/workflows/blueprints/test_blueprints.py

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -160,22 +160,32 @@ async def test_get_blueprint_details(self, mcp_client):
160160

161161
async def test_get_blueprint_not_found(self, mcp_client):
162162
"""
163-
Test: Get blueprint that doesn't exist
163+
Test: ha_get_blueprint with a nonexistent path returns a structured
164+
error with code RESOURCE_NOT_FOUND, not success=True.
164165
165-
Validates proper error handling when blueprint path doesn't exist.
166+
Source path: tools_blueprints.py — when the requested path is absent
167+
from the blueprints registry, raise_tool_error is invoked with
168+
ErrorCode.RESOURCE_NOT_FOUND and the message "Blueprint not found: ...".
169+
170+
Hardened from a single suggestions-presence check to explicit
171+
error-code and structured suggestion-presence assertions.
166172
"""
167173
logger.info("Testing ha_get_blueprint with non-existent path...")
168174

169175
async with MCPAssertions(mcp_client) as mcp:
170176
# Try to get a non-existent blueprint
171177
result = await mcp.call_tool_failure(
172178
"ha_get_blueprint",
173-
{"path": "nonexistent/blueprint_xyz.yaml", "domain": "automation"},
179+
{"path": "nonexistent/blueprint_a2_e2e_xyz_404.yaml", "domain": "automation"},
174180
expected_error="not found",
175181
)
176182

177-
# Verify error response includes suggestions (nested under "error")
178-
assert "suggestions" in result.get("error", {}), "Error response should include suggestions"
183+
assert result["error"]["code"] == "RESOURCE_NOT_FOUND", (
184+
f"Expected error code RESOURCE_NOT_FOUND, got: {result['error']}"
185+
)
186+
assert "suggestion" in result["error"], (
187+
"Error response should include a suggestion"
188+
)
179189
logger.info("ha_get_blueprint properly handles non-existent blueprint")
180190

181191
async def test_get_blueprint_invalid_domain(self, mcp_client):

tests/src/e2e/workflows/config/test_label_crud.py

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -202,18 +202,39 @@ async def test_create_label_with_icon(self, mcp_client, cleanup_tracker):
202202
)
203203

204204
async def test_get_nonexistent_label(self, mcp_client):
205-
"""Test getting a non-existent label."""
205+
"""
206+
Test: ha_config_get_label with a nonexistent label_id returns a
207+
structured error with code ENTITY_NOT_FOUND, not success=True.
208+
209+
Source path: tools_labels.py — after listing labels via WebSocket,
210+
the requested label_id is looked up in the result. When absent,
211+
raise_tool_error is invoked with ErrorCode.ENTITY_NOT_FOUND and the
212+
message "Label not found: ...".
213+
214+
Hardened from success-only check to explicit error-code and
215+
message-substring assertions.
216+
"""
206217
logger.info("Testing get non-existent label")
207218

208219
data = await safe_call_tool(
209220
mcp_client,
210221
"ha_config_get_label",
211-
{"label_id": "nonexistent_label_xyz_12345"},
222+
{"label_id": "nonexistent_label_a2_e2e_xyz_404"},
212223
)
213224

214225
assert data.get("success") is False, (
215226
f"Should fail for non-existent label: {data}"
216227
)
228+
assert data["error"]["code"] == "ENTITY_NOT_FOUND", (
229+
f"Expected error code ENTITY_NOT_FOUND, got: {data['error']}"
230+
)
231+
assert "suggestion" in data["error"], (
232+
"Error response should include a suggestion"
233+
)
234+
error_msg = data["error"]["message"].lower()
235+
assert "not found" in error_msg, (
236+
f"Expected 'not found' in error message, got: {data['error']}"
237+
)
217238
logger.info("Non-existent label properly returned error")
218239

219240
async def test_delete_nonexistent_label(self, mcp_client):

tests/src/e2e/workflows/integrations/test_list_integrations.py

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212

1313
import pytest
1414

15-
from ...utilities.assertions import assert_mcp_success
15+
from ...utilities.assertions import assert_mcp_success, safe_call_tool
1616

1717
logger = logging.getLogger(__name__)
1818

@@ -421,3 +421,49 @@ async def test_integration_discovery(mcp_client):
421421
logger.info(
422422
f"Integration discovery test passed: found {data['total_count']} integrations"
423423
)
424+
425+
426+
@pytest.mark.integrations
427+
class TestGetIntegrationNegativeInputs:
428+
"""
429+
A2 negative-input tests for ha_get_integration's single-entry lookup mode.
430+
431+
Covers the nonexistent-entry_id failure path (entry_id provided, no match
432+
in the config-entry registry). The existing tests in this file exercise
433+
the list/query/domain-filter modes, which return empty results rather
434+
than errors — that is A4-style behavior. The entry_id branch goes through
435+
a distinct error path and was previously untested.
436+
437+
Methodology: source-verified against tools_integrations.py. When
438+
_get_single_entry encounters a 404 from the underlying REST/WebSocket
439+
call, raise_tool_error is invoked with ErrorCode.RESOURCE_NOT_FOUND and
440+
the message "Config entry not found: ...".
441+
"""
442+
443+
async def test_get_integration_nonexistent_entry_id(self, mcp_client):
444+
"""
445+
Test: ha_get_integration(entry_id="<nonexistent>") returns a
446+
structured error with code RESOURCE_NOT_FOUND, not success=True.
447+
448+
Source path: tools_integrations.py — _get_single_entry catches a
449+
404/not-found exception and raises RESOURCE_NOT_FOUND.
450+
"""
451+
data = await safe_call_tool(
452+
mcp_client,
453+
"ha_get_integration",
454+
{"entry_id": "nonexistent_entry_a2_e2e_xyz_404"},
455+
)
456+
457+
assert not data.get("success"), (
458+
f"Expected failure for nonexistent entry_id, got success=True: {data}"
459+
)
460+
assert data["error"]["code"] == "RESOURCE_NOT_FOUND", (
461+
f"Expected error code RESOURCE_NOT_FOUND, got: {data['error']}"
462+
)
463+
assert "suggestion" in data["error"], (
464+
"Error response should include a suggestion"
465+
)
466+
error_msg = data["error"]["message"].lower()
467+
assert "not found" in error_msg, (
468+
f"Expected 'not found' in error message, got: {data['error']}"
469+
)

tests/src/e2e/workflows/registry/test_device_registry.py

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -625,3 +625,49 @@ async def test_device_entity_independence(mcp_client):
625625
assert restore_data.get("success"), f"Failed to restore device name: {restore_data}"
626626

627627
logger.info("Device/entity naming independence test completed")
628+
629+
630+
@pytest.mark.registry
631+
class TestDeviceGetNegativeInputs:
632+
"""
633+
A2 negative-input tests for ha_get_device's single-device lookup mode.
634+
635+
Covers the nonexistent-device_id failure path. Existing tests in this
636+
file exercise the list mode, area/manufacturer filters, and the
637+
update/remove flows, but never call ha_get_device with a device_id
638+
that is absent from the device registry.
639+
640+
Methodology: source-verified against tools_registry.py. When the
641+
requested device_id is not present in the device registry list,
642+
raise_tool_error is invoked with ErrorCode.ENTITY_NOT_FOUND and the
643+
message "Device not found: ...".
644+
"""
645+
646+
async def test_get_device_nonexistent_device_id(self, mcp_client):
647+
"""
648+
Test: ha_get_device(device_id="<nonexistent>") returns a structured
649+
error with code ENTITY_NOT_FOUND, not success=True.
650+
651+
Source path: tools_registry.py — single-device lookup branch returns
652+
ENTITY_NOT_FOUND when the device_id is absent from
653+
config/device_registry/list.
654+
"""
655+
data = await safe_call_tool(
656+
mcp_client,
657+
"ha_get_device",
658+
{"device_id": "nonexistent_device_a2_e2e_xyz_404"},
659+
)
660+
661+
assert not data.get("success"), (
662+
f"Expected failure for nonexistent device_id, got success=True: {data}"
663+
)
664+
assert data["error"]["code"] == "ENTITY_NOT_FOUND", (
665+
f"Expected error code ENTITY_NOT_FOUND, got: {data['error']}"
666+
)
667+
assert "suggestion" in data["error"], (
668+
"Error response should include a suggestion"
669+
)
670+
error_msg = data["error"]["message"].lower()
671+
assert "not found" in error_msg, (
672+
f"Expected 'not found' in error message, got: {data['error']}"
673+
)

tests/src/e2e/workflows/updates/test_updates.py

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -534,3 +534,45 @@ async def test_update_tools_discovery(mcp_client):
534534
)
535535

536536
logger.info("Update tools discovery test passed")
537+
538+
539+
@pytest.mark.updates
540+
class TestUpdatesGetNegativeInputs:
541+
"""
542+
A2 negative-input tests for ha_get_updates' single-entity detail mode.
543+
544+
Covers the nonexistent-entity_id failure path. Existing tests in this
545+
file exercise listing, release-notes inclusion, and edge cases on real
546+
update entities, but do not call ha_get_updates with an entity_id that
547+
has no matching update.
548+
549+
Methodology: source-verified against tools_updates.py. _get_update_details
550+
fetches the entity state via REST; a 404 from /api/states/<entity_id>
551+
raises HomeAssistantAPIError("API error: 404 - Entity not found.").
552+
The except-Exception branch in ha_get_updates matches "404" / "not found"
553+
in the error message and raises ErrorCode.ENTITY_NOT_FOUND. Live-probed
554+
against a real HA instance: GET /api/states/update.<nonexistent> returns
555+
HTTP 404 with body {"message": "Entity not found."}.
556+
"""
557+
558+
async def test_get_updates_nonexistent_entity_id(self, mcp_client):
559+
"""
560+
Test: ha_get_updates(entity_id="update.<nonexistent>") returns a
561+
structured error with code ENTITY_NOT_FOUND, not success=True.
562+
563+
Source path: REST 404 → HomeAssistantAPIError → except-Exception
564+
in ha_get_updates → ENTITY_NOT_FOUND.
565+
"""
566+
async with MCPAssertions(mcp_client) as mcp:
567+
data = await mcp.call_tool_failure(
568+
"ha_get_updates",
569+
{"entity_id": "update.nonexistent_a2_e2e_xyz_404"},
570+
expected_error="not found",
571+
)
572+
573+
assert data["error"]["code"] == "ENTITY_NOT_FOUND", (
574+
f"Expected error code ENTITY_NOT_FOUND, got: {data['error']}"
575+
)
576+
assert "suggestion" in data["error"], (
577+
"Error response should include a suggestion"
578+
)

0 commit comments

Comments
 (0)