Skip to content

Commit 033a80b

Browse files
author
Task Orchestrator Agent
committed
test(visual): add Playwright E2E visual regression suite for critical flows
Add a dedicated visual-regression test suite (toHaveScreenshot) covering the critical NodeTool user flows, complementing the documentation-screenshot suite (which only writes PNGs) with committed baselines that fail on pixel diffs. Coverage (21 unique page states across 4 projects / 39 instances): - Node Graph Editor: empty canvas, nodes+edges, inspector, node library, timeline - Chat: empty thread, message thread, media composer, model selector, dashboard - Settings: API Keys (provider cards), Integrations, General, About - Design system: color picker, recommended-models, image comparer, layout primitives - Theme: dashboard/chat/settings captured in light mode (dark is default elsewhere) Projects: desktop-chromium (1440), mobile-chromium (375), tablet-chromium (768), firefox-desktop (cross-browser smoke). Tag-based selection (@responsive/@smoke). Determinism: real seeded in-memory backend (reuses tests/globalSetup.ts), theme pinned via localStorage before paint, CSS animations/transitions frozen via init script, onboarding+panels seeded, single worker, no retries, 1% diff tolerance. CI (.github/workflows/visual-regression.yml): runs on PRs touching web/src or the visual suite; builds packages, installs chromium+firefox, runs the suite, uploads baselines+report. A workflow_dispatch update=true job regenerates and commits baselines. Missing baselines auto-create and pass (Playwright "missing" mode); committed baselines enforce diffs. Docs: web/tests/visual/README.md covers running, updating baselines (local + CI), and stability techniques. typecheck:visual / test:visual[:update] npm scripts added. Note: baselines are not committed in this change — they auto-generate on the first CI run (and are downloadable as an artifact) or via the update-baselines dispatch. Local empirical verification was blocked by the shared prewarm node_modules (no built dist) and missing Chromium system libs in this runner; the suite reuses the proven patterns from tests/benchmarks/screenshots.spec.ts.
1 parent adb9e51 commit 033a80b

12 files changed

Lines changed: 1238 additions & 2 deletions

‎.github/workflows/visual-regression.yml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
name: Visual Regression
2+
3+
on:
4+
pull_request:
5+
paths:
6+
# Run when the visual suite, config, or the web UI it screenshots changes.
7+
# Backend package-only changes are verified via the dispatch workflow.
8+
- "web/tests/visual/**"
9+
- "web/playwright.visual.config.ts"
10+
- "web/src/**"
11+
- "web/package.json"
12+
- ".github/workflows/visual-regression.yml"
13+
push:
14+
branches: [main]
15+
paths:
16+
- "web/tests/visual/**"
17+
- "web/playwright.visual.config.ts"
18+
- "web/src/**"
19+
workflow_dispatch:
20+
inputs:
21+
update:
22+
description: "Regenerate baselines and commit them to the branch"
23+
required: false
24+
default: "false"
25+
type: choice
26+
options: ["false", "true"]
27+
28+
permissions:
29+
contents: write # so the update-baselines job can push back to the branch
30+
31+
jobs:
32+
visual:
33+
name: Playwright visual snapshots
34+
runs-on: ubuntu-latest
35+
# Don't run the auto-update path on pull_request — only via dispatch or push
36+
# with inputs.update=true. The comparison run is skipped when updating.
37+
if: ${{ !(github.event.inputs.update == 'true') }}
38+
39+
steps:
40+
- name: Checkout code
41+
uses: actions/checkout@v7
42+
43+
- name: Set up Node.js
44+
uses: actions/setup-node@v6
45+
with:
46+
node-version-file: ".nvmrc"
47+
cache: "npm"
48+
49+
- name: Install dependencies
50+
run: npm ci
51+
52+
- name: Build workspace packages
53+
# The seeded screenshot backend (tests/globalSetup.ts) imports
54+
# @nodetool-ai/* via the `development` export condition, which resolves
55+
# to each package's compiled dist/. Build them first, same as the
56+
# documentation-screenshot workflow.
57+
run: npm run build:packages
58+
59+
- name: Install Playwright browsers (chromium + firefox)
60+
working-directory: web
61+
run: npx playwright install chromium firefox --with-deps
62+
63+
- name: Run visual regression tests
64+
working-directory: web
65+
env:
66+
# Same deterministic test-only master key committed in
67+
# web/tests/globalSetup.ts. Never use in production.
68+
SECRETS_MASTER_KEY: "U0NSRUVOU0hPVF9URVNUX0tFWV9ET19OT1RfVVNFISE="
69+
run: npm run test:visual
70+
71+
- name: Upload visual baselines + report
72+
uses: actions/upload-artifact@v7
73+
if: always()
74+
with:
75+
name: visual-regression
76+
# Baselines (to download + commit on first run) and the HTML report
77+
# with side-by-side diffs for any failures.
78+
path: |
79+
web/tests/visual/**/*-snapshots/
80+
web/playwright-report/
81+
retention-days: 30
82+
83+
update-baselines:
84+
name: Update visual baselines
85+
runs-on: ubuntu-latest
86+
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.update == 'true' }}
87+
steps:
88+
- name: Checkout code
89+
uses: actions/checkout@v7
90+
with:
91+
ref: ${{ github.ref }}
92+
93+
- name: Set up Node.js
94+
uses: actions/setup-node@v6
95+
with:
96+
node-version-file: ".nvmrc"
97+
cache: "npm"
98+
99+
- name: Install dependencies
100+
run: npm ci
101+
102+
- name: Build workspace packages
103+
run: npm run build:packages
104+
105+
- name: Install Playwright browsers
106+
working-directory: web
107+
run: npx playwright install chromium firefox --with-deps
108+
109+
- name: Regenerate baselines
110+
working-directory: web
111+
env:
112+
SECRETS_MASTER_KEY: "U0NSRUVOU0hPVF9URVNUX0tFWV9ET19OT1RfVVNFISE="
113+
run: npm run test:visual:update
114+
115+
- name: Commit updated baselines
116+
run: |
117+
git config user.name "github-actions[bot]"
118+
git config user.email "github-actions[bot]@users.noreply.github.com"
119+
git add web/tests/visual/**/*-snapshots/
120+
if git diff --cached --quiet; then
121+
echo "No baseline changes to commit."
122+
else
123+
git commit -m "chore(visual): update E2E visual baselines [skip ci]"
124+
git push
125+
fi

‎web/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,10 @@
104104
"test:e2e-runner:headed": "playwright test -c playwright.e2e-runner.config.ts --headed",
105105
"test:debug-harness": "playwright test -c playwright.debug-harness.config.ts",
106106
"e2e-suite:prepare": "tsx tests/e2e-runner/prepareSuite.ts",
107+
"test:visual": "playwright test -c playwright.visual.config.ts",
108+
"test:visual:headed": "playwright test -c playwright.visual.config.ts --headed",
109+
"test:visual:update": "playwright test -c playwright.visual.config.ts --update-snapshots",
110+
"typecheck:visual": "tsc --noEmit -p tsconfig.visual.json",
107111
"typecheck": "tsc --noEmit",
108112
"upload-stock-images": "node scripts/upload-stock-images.js",
109113
"screenshots": "playwright test tests/benchmarks/screenshots.spec.ts --project=chromium",

‎web/playwright.config.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,10 @@ import { defineConfig, devices } from "@playwright/test";
2929
export default defineConfig({
3030
testDir: "./tests",
3131

32-
/* The E2E workflow runner has its own config + backend (e2e-server). */
33-
testIgnore: "**/e2e-runner/**",
32+
/* The E2E workflow runner and the visual-regression suite each have their
33+
own config + project selection; exclude them from this (documentation
34+
screenshot) config so `npx playwright test` doesn't pick them up. */
35+
testIgnore: ["**/e2e-runner/**", "**/visual/**"],
3436

3537
/* Maximum time one test can run */
3638
timeout: 60_000,

‎web/playwright.visual.config.ts‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
import { defineConfig, devices } from "@playwright/test";
2+
3+
/**
4+
* Playwright configuration for E2E visual regression tests.
5+
*
6+
* These tests assert against committed screenshot baselines via
7+
* `expect(page).toHaveScreenshot(...)` and fail on unexpected pixel diffs.
8+
* They complement the documentation screenshot suite
9+
* (tests/benchmarks/screenshots.spec.ts) which only writes PNGs for the docs.
10+
*
11+
* What it exercises (critical user flows):
12+
* - Node Graph Editor: empty canvas, nodes+edges, inspector, timeline
13+
* - Chat Interface: empty thread, message thread, media composer, menus
14+
* - Settings: API Keys + Integrations tabs and provider cards
15+
* - Design system: isolated component previews (component gallery)
16+
*
17+
* Determinism:
18+
* - The real NodeTool backend is started once by `tests/globalSetup.ts` with
19+
* an in-memory SQLite DB pre-seeded with fixed mock data (no clock, no
20+
* network). Vite serves the frontend (auto-started below) and proxies
21+
* /api + /ws to the backend on :7777.
22+
* - Theme is pinned via localStorage; CSS transitions/animations are frozen
23+
* via an init script (see tests/visual/visualHelpers.ts).
24+
* - Single worker, no retries — visual diffs must be deterministic.
25+
*
26+
* Projects (browser × viewport):
27+
* - desktop-chromium (1440×900) → runs every visual test
28+
* - mobile-chromium (375×812) → @responsive tests only
29+
* - tablet-chromium (768×1024) → @responsive tests only
30+
* - firefox-desktop (1440×900) → @smoke tests only (cross-browser guard)
31+
*
32+
* Run:
33+
* npx playwright test --config=playwright.visual.config.ts
34+
* npx playwright test --config=playwright.visual.config.ts --update-snapshots
35+
* npm run test:visual # alias for the first command
36+
* npm run test:visual:update # regenerate baselines
37+
*
38+
* Baselines live next to each spec under `<spec>.spec.ts-snapshots/` and are
39+
* committed. See tests/visual/README.md for the full update workflow.
40+
*/
41+
export default defineConfig({
42+
testDir: "./tests/visual",
43+
testMatch: /.*\.spec\.ts$/,
44+
45+
/* One test can take a while (backend round-trips + canvas layout). */
46+
timeout: 60_000,
47+
48+
/* Visual assertions compare many pixels — give them room. */
49+
expect: {
50+
timeout: 15_000,
51+
toHaveScreenshot: {
52+
// Sub-pixel font-hinting / anti-aliasing differs slightly between
53+
// Chromium and Firefox and across headless renderers. A 1% tolerance
54+
// absorbs that noise while still failing on real layout regressions.
55+
maxDiffPixelRatio: 0.01,
56+
threshold: 0.2
57+
}
58+
},
59+
60+
forbidOnly: !!process.env.CI,
61+
/* No retries — a flaky visual diff should be fixed, not hidden. */
62+
retries: 0,
63+
/* Sequential execution: deterministic + the single shared backend is not
64+
designed for concurrent browsers hammering it at once. */
65+
workers: 1,
66+
67+
reporter: process.env.CI ? "github" : "list",
68+
69+
/** Reuse the same seeded real backend the screenshot suite uses. */
70+
globalSetup: "./tests/globalSetup.ts",
71+
72+
use: {
73+
baseURL: "http://localhost:3000",
74+
trace: "retain-on-failure",
75+
screenshot: "only-on-failure",
76+
ignoreHTTPSErrors: true
77+
},
78+
79+
projects: [
80+
{
81+
name: "desktop-chromium",
82+
use: {
83+
...devices["Desktop Chrome"],
84+
viewport: { width: 1440, height: 900 },
85+
launchOptions: {
86+
...(process.env.PLAYWRIGHT_CHROMIUM_PATH
87+
? { executablePath: process.env.PLAYWRIGHT_CHROMIUM_PATH }
88+
: {}),
89+
args: [
90+
"--enable-features=Vulkan,UseSkiaRenderer",
91+
"--use-gl=angle",
92+
"--use-angle=swiftshader",
93+
"--enable-webgpu-developer-features"
94+
]
95+
}
96+
}
97+
},
98+
{
99+
name: "mobile-chromium",
100+
grep: /@responsive/,
101+
use: {
102+
...devices["Desktop Chrome"],
103+
viewport: { width: 375, height: 812 },
104+
isMobile: true,
105+
hasTouch: true,
106+
launchOptions: {
107+
...(process.env.PLAYWRIGHT_CHROMIUM_PATH
108+
? { executablePath: process.env.PLAYWRIGHT_CHROMIUM_PATH }
109+
: {}),
110+
args: [
111+
"--enable-features=Vulkan,UseSkiaRenderer",
112+
"--use-gl=angle",
113+
"--use-angle=swiftshader",
114+
"--enable-webgpu-developer-features"
115+
]
116+
}
117+
}
118+
},
119+
{
120+
name: "tablet-chromium",
121+
grep: /@responsive/,
122+
use: {
123+
...devices["Desktop Chrome"],
124+
viewport: { width: 768, height: 1024 },
125+
launchOptions: {
126+
...(process.env.PLAYWRIGHT_CHROMIUM_PATH
127+
? { executablePath: process.env.PLAYWRIGHT_CHROMIUM_PATH }
128+
: {}),
129+
args: [
130+
"--enable-features=Vulkan,UseSkiaRenderer",
131+
"--use-gl=angle",
132+
"--use-angle=swiftshader",
133+
"--enable-webgpu-developer-features"
134+
]
135+
}
136+
}
137+
},
138+
{
139+
name: "firefox-desktop",
140+
// Cross-browser smoke guard: a small, stable subset. Firefox does not
141+
// initialise the WebGPU canvases the same way Chromium does, so we keep
142+
// it to pages that don't depend on them.
143+
grep: /@smoke/,
144+
use: {
145+
...devices["Desktop Firefox"],
146+
viewport: { width: 1440, height: 900 }
147+
}
148+
}
149+
],
150+
151+
/**
152+
* Auto-start Vite. The dev server proxies /api/* and /ws to the backend on
153+
* :7777 started by globalSetup. Reuse an already-running server locally so
154+
* `npm run dev` + `npm run test:visual` don't fight over port 3000.
155+
*/
156+
webServer: {
157+
command: "npm start",
158+
url: "http://localhost:3000",
159+
reuseExistingServer: !process.env.CI,
160+
timeout: 120_000
161+
}
162+
});

0 commit comments

Comments
 (0)