Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .agent/skills/testmanagement-create-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ metadata:

Follow these instructions when the user wants to create a new manual functional test case through the `/test-create` workflow.

## General Test Generation Rules

- **Randomized Test Data**: Never use fixed or hardcoded test data (like specific product names, static URLs, or hardcoded dimensions) in the test case instructions unless explicitly required to test a unique edge case. Always instruct the tester to use `[Randomized]` or parametrical placeholders. This guarantees the drafted tests are durable and not brittle to future catalog or price changes.

Ask the user if they want to execute a **Light Flow** or a **Standard Flow** when this skill is invoked.

## Light Flow
Expand All @@ -22,6 +26,7 @@ Ask the user if they want to execute a **Light Flow** or a **Standard Flow** whe
5. Draft the new test case inside the appropriate domain directory (e.g., `doc/tests/checkout/TC_CHK_...`).
6. Ensure Metadata, Preconditions, Steps, Execution Targets, and Change History are perfectly populated.
7. Present the draft to the user and conduct a fast review round to refine and improve the test case. Once accepted, update the domain's `overview.md` with the new test item.
8. **Optional Validation:** Ask the user if they want to validate the physical test case using the `browser_subagent` AND ask them to provide the target URL (e.g., localhost, staging). If they provide it, navigate to the URL and follow the drafted Markdown exactly to prove structural correctness.

## Standard Flow

Expand All @@ -36,3 +41,4 @@ You must act as the orchestrator of the exact 9-step QA process specified in the
7. **Test Manager Review:** Act as a Test Manager to verify coverage, traceability, and execution efficiency.
8. **End Consumer Review:** Act as an End Consumer to verify usability, clarity, and real-world alignment.
9. **Finalize:** Automatically update the domain's `overview.md` with the new test item, and require explicit sign-off from the user before concluding the domain creation.
10. **Optional Validation:** As a final phase, ask the user if they'd like to physically validate the test using the `browser_subagent`. If so, explicitly ask them for the target validation URL, then load the site and step through the test to formally ensure structural accuracy before considering the feature stable.
1 change: 1 addition & 0 deletions .agent/workflows/test-create.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,6 @@ Help the user brainstorm and generate structured manual test cases in the `doc/t
When the user runs `/test-create`, I will:
1. Ask them about the domain, feature requirements, and workflows they want to test.
2. Draft the requested test case(s) using the official `TEMPLATE.md` formats and place them in the correct directories.
3. Offer an optional physical test validation pass using the AI `browser_subagent` to ensure structural layout alignment.

For specific behaviors, I must use the instructions in `.agent/skills/testmanagement-create-tests/SKILL.md`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-04-17
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## Context

The Posters Demo Store utilizes AI agents to autonomously draft manual QA test cases that human or automated testers can later follow. Currently, these tests are generated strictly from static W3C flows and templates. Because there's no feedback loop enforcing that the AI actually validated the selectors, URL paths, and exact behavioral logic against the live site, the agent risks producing test specifications that contain hallucinations.

## Goals / Non-Goals

**Goals:**
- Guarantee physical validation of all AI-drafted test cases prior to their finalization.
- Automatically invoke the `browser_subagent` as part of the `testmanagement-create-tests` workflow to step through the draft test.
- Create a reusable standard for an `agent-browser-validation` capability.

**Non-Goals:**
- This is not a complete end-to-end automated UI framework replacing Playwright. It's strictly the usage of the agent subagent to validate structural logic and text accuracy during the creation/draft phase of manual test cases.

## Decisions

- **Workflow Integration**: The browser validation step will be injected into both the `Light Flow` (after drafting) and `Standard Flow` (by appending it as 'Step 10'). However, to support scenarios where the system is not yet deployed but tests need to be prepared (e.g., from mockups), the agent will *ask* the user if they want to execute the validation step. It will be an optional mode.
- **Standalone Capability Spec**: The OpenSpec structure will introduce this logic as `openspec/specs/agent-browser-validation/spec.md` to cleanly separate meta agent toolsets from standard commerce product logic.

## Risks / Trade-offs

- **Performance Trade-off**: Enforcing a browser subagent trace linearly slows down the time it takes the Agent to report success back to the end user. This trade-off is absolutely worth the increase in 100% accurate test definitions.
- **Flaky Environmental Behavior**: The `browser_subagent` operations could fail due to timing or element loading even when the test steps are technically valid. Agent instructions must ensure graceful recovery and explicit error logs indicating if the failure was a structural hallucination or a system timeout.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## Why

When the agent uses the `/test-create` workflow to generate manual QA test cases, it drafts the markdown based on instructions and expected W3C flows. However, without executing the test, the generated steps could be hallucinated, mismatch the physical UI, or reference non-existent elements. Adding a synchronized requirement where the agent *physically* validates the test case using the browser subagent guarantees that all committed test cases are factually working against the application.

## What Changes

We are extending the `testmanagement-create-tests` skill and `/test-create` workflow to inject a final (or late-stage) execution step. During this step, the agent will load the locally running application using the `browser_subagent` and walk through the dynamically generated test case steps. Only if the validation succeeds will the test case be structurally signed-off and the domain initialized.

## Capabilities

### New Capabilities

- `agent-browser-validation`: Defines the capability of an AI agent to structurally launch a browser subagent and autonomously validate user-facing flows according to a defined Markdown test specification prior to saving.

### Modified Capabilities

- No existing OpenSpec capabilities are modified (the actual change happens in the agent architecture `.agent/skills/` and `.agent/workflows/` which are not currently modeled as formal OpenSpecs).

## Impact

- `.agent/skills/testmanagement-create-tests/SKILL.md`
- `.agent/workflows/test-create.md`
- System reliability of generated domain test cases.
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Implementation Tasks

## 1. Agent Skill File Update
- **Target File**: `.agent/skills/testmanagement-create-tests/SKILL.md`
- **Action**: Update the instructions for both Light and Standard flows.
- In `Light Flow`, state that the agent should offer an optional physical browser validation of the formulated test case using `browser_subagent`.
- In `Standard Flow`, append an optional `Step 10: Automated Browser Validation` which acts as a final execution check if the product is live.

## 2. Agent Workflow Update
- **Target File**: `.agent/workflows/test-create.md`
- **Action**: Add an explanation in the overview that the agent supports an optional browser validation loop.
14 changes: 14 additions & 0 deletions openspec/specs/agent-browser-validation/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Agent Browser Validation

Enable AI agents to automatically launch a localized browser instance to physically trace and validate manually drafted test cases against a live environment, ensuring the test specification structurally matches the actual application.

## Requirements

### Workflow Integration
- **Optional Execution & Target URL**: The browser validation must be an **optional** capability. The agent must prompt the user before execution and request the target environment URL (e.g., asking "Do you want me to physically validate this test? If yes, please provide the target URL."). This ensures tests can be drafted without an active deployment and allows validation against any environment (local, staging, prod).
- **Light Flow**: After drafting a test case and during the final refinement, the agent offers the automated browser validation mode.
- **Standard Flow**: The strict 9-step test creation workflow is expanded with an optional "Step 10: Automated Browser Validation".

### System Traceability
- The agent must use the `browser_subagent` skill to strictly execute the `Action` and `Data` flags as defined in the drafted Markdown document, observing the exact `Verify` parameters.
- Any physical deviation, unresponsive element, or missing layout state must be treated as a validation failure. The agent must surface this to the user for collaborative correction of the test case.
Loading