Skip to content

feat: post-deploy E2E browser verification step (issue → spec → implement → review → merge → deploy → E2E verify) #604

Description

@baderdean

Problem

Looper's loop currently ends at merge. But many projects (including those with AGENTS.md mandates) require verifying the deployed result after merge — not just the diff. For example, an m3llm AGENTS.md mandates a browser E2E test on https://m3llm.cafe after every user-facing feature:

  • Navigate to the deployed URL
  • Type into the chat input
  • Click submit
  • Verify the tool is invoked and returns data
  • Screenshot the result
  • At least 2 use cases (happy + edge)

Today the reviewer correctly marks the 405 fix as UNVERIFIABLE because it's a deployment issue (Edge Function not deployed), not a code issue. The code is correct in the diff. But the 405 IS verifiable post-deploy by hitting the live endpoint. Looper has no step for this.

Proposal

Add an optional post-deploy E2E verification step to the loop:

issue → planner → spec PR → review → worker → impl PR → review → merge → [deploy] → E2E verify

The E2E step would:

  1. Wait for deploy to complete (configurable: GitHub Actions workflow, CF Pages deploy, manual signal, or fixed delay)
  2. Run a browser automation against the deployed URL (configurable: Playwright, CF Browser Rendering, agent-browser skill, or custom command)
  3. Verify acceptance criteria from the original issue (the agent reads the issue + spec, derives test cases, runs them)
  4. Post results as a comment on the original issue (with screenshots)
  5. If E2E fails: reopen the issue OR open a new bug issue with the failure evidence

Why this matters

  • The loop currently has a gap between merge and verified-in-production
  • Reviewers correctly mark deployment-dependent fixes as UNVERIFIABLE — but they ARE verifiable, just not from the diff
  • AGENTS.md mandates (browser E2E on shared fixtures) are common in real projects
  • baton (mraza007/baton) already does this: agent-browser E2E verifies acceptance criteria before PR. But looper's loop ends at merge, not before.
  • Cursor Background Agents use video recording + screenshots as verification — turns diff into proof

Design considerations

  • Harness-agnostic: the browser automation should be pluggable (Playwright, CF Browser Rendering, agent-browser skill, custom command)
  • Configurable trigger: not every PR needs E2E (docs, refactors don't). Could be gated by label (looper:e2e), issue category, or AGENTS.md directive
  • Deploy detection: GitHub Actions workflow run status, CF Pages deployment status, webhook, or fixed delay
  • Failure handling: reopen issue, open new bug, or just comment (configurable)
  • Cost: browser E2E is expensive — should be optional and budget-controlled

Out of scope

Context

  • m3llm AGENTS.md mandates browser E2E on m3llm.cafe after deploy (3-30min)
  • baton's agent-browser E2E: open/snapshot/click/fill/type to verify acceptance criteria
  • Cursor Background Agents: video recording + screenshots as verification
  • CF Browser Rendering binding could serve as a serverless browser backend

Environment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions