Skip to content

Commit b2cc1b4

Browse files
mmckyclaudegithub-actions[bot]
authored
Migrate visual regression tests to quantecon-book-theme-fixtures (#390)
* Migrate visual regression tests to quantecon-book-theme-fixtures Replace the lecture-python-programming clone (moving target, conda build, ~3-5 min per CI run) with the curated quantecon-book-theme-fixtures repo pinned at SHA d8ffc17. The fixtures site builds in ~30-60s with pip-only installs, has stable inputs across PRs, and covers more theme surfaces (definition lists, deeply nested TOC, etc.) than the previous lecture subset could. Changes: - ci.yml + update-snapshots.yml: checkout fixtures repo at FIXTURES_SHA, pip-install requirements + this PR's theme, jb build --warningiserror, point Playwright at fixtures/_build/html - tox.ini: visual + visual-update envs clone the fixtures repo (FIXTURES_REF env var lets you pin a SHA locally) - playwright.config.ts: default SITE_PATH points at fixtures/_build/html - theme.spec.ts: rewrite page list for 13 synthetic + 1 real-world fixture; add definition-list tests (previously skipped — no <dl> in the lecture site); drop hasPlots distinction (fixtures don't execute notebooks) - tests/visual/README.md: rewritten around fixtures workflow - .gitignore: replace lecture-python-programming entry with fixtures - Delete stale ubuntu snapshots under tests/visual/__snapshots__/ — page paths changed; comment /update-new-snapshots on this PR to seed fresh baselines The /update-new-snapshots and /update-snapshots PR comment commands still work the same way; they now build the fixtures site instead of the lecture site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address Copilot review feedback on #390 - tox.ini: clone fixtures without --branch, then fetch + checkout ${FIXTURES_REF:-main} on every run. Previously --branch only accepted branch/tag names (not SHAs, despite the README documenting SHA usage) and the [ -d fixtures ] guard meant FIXTURES_REF changes were silently ignored on subsequent runs. - tox.ini: always run jb build (sphinx is incremental). The previous skip-if-built guard caused tox -e visual to serve stale theme assets when the theme changed between runs — CI was unaffected but local dev could silently test against the old theme. - theme.spec.ts: gate the loose maxDiffPixelRatio: 0.05 on a per-page hasMath flag. Non-math pages now keep the strict default tolerance from playwright.config.ts so sub-pixel regressions still get caught. - tests/visual/README.md: clarify that the test set is the landing page + 12 synthetic pages + real-world captures (was: "12 synthetic pages" alone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update visual-testing.md for fixtures-based workflow Missed in the initial migration commit — the user-facing developer docs at docs/developer/visual-testing.md still described the lecture-python-programming flow (clone + jb build lectures + …). Rewritten to match the rest of the PR: fixtures clone, FIXTURES_SHA pinning, FIXTURES_REF for local pins, and the issue_comment + default-branch gotcha that motivated the #392 split. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * UPDATE: Visual regression snapshots (new) * Bump toHaveScreenshot timeout to 30s for long-page fixtures The /update-new-snapshots run on #390 successfully generated baselines for 13 of 14 fixture pages, but prob-matrix full-page screenshots timed out at Playwright's default 5000ms on both desktop and mobile — prob_matrix.md is ~1900 source lines, much taller than the other fixtures, and Playwright couldn't stitch the full-page capture in time. Header and sidebar baselines for prob-matrix were generated fine. Bumping the toHaveScreenshot timeout to 30s gives long-page captures plenty of headroom while leaving short captures unaffected (they still finish in well under a second). After this lands, re-comment /update-new-snapshots on this PR to seed the two missing prob-matrix baselines (workflow leaves existing baselines alone, only creates missing ones). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move screenshot timeout to per-assertion call (real fix) The previous attempt put timeout: 30000 in expect.toHaveScreenshot in playwright.config.ts, but timeout isn't a valid key in that config block (Playwright silently ignores it — the latest /update-new-snapshots run still hit "Timeout 5000ms exceeded" on prob-matrix). The valid places are expect.timeout (too broad — affects all matchers) or per-assertion. Adding timeout: 30000 directly to the full-page toHaveScreenshot call in the spec, which is the canonical Playwright pattern. Reverting the no-op config change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * UPDATE: Visual regression snapshots (new) * ci: trigger run on latest branch state The previous CI run was on c6d3a60 (the timeout fix), before the bot's snapshot commit 4d48a02 landed. Commits made by GITHUB_TOKEN don't trigger downstream workflows by design, so the baselines committed by the /update-new-snapshots bot never got a CI pass. This empty commit forces a fresh run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Disable full-page screenshot test for prob-matrix fixture The CI run after the bot's baseline commit revealed that the prob-matrix full-page capture isn't stable: expected height 41452px, captured height 40693px on subsequent runs (759px difference). Playwright requires identical image dimensions for comparison, so no tolerance setting can help. The variance comes from MathJax font-metric resolution + image loading + table layout — networkidle + MathJax.startup.promise resolves before the page actually stops growing. Real-world fixtures inherently have this characteristic; the synthetic math.html page already provides full-page math regression coverage with stable dimensions. For prob-matrix, header + sidebar region tests still run (those are stable theme elements) and the Netlify preview still renders the full page for human visual review — which is the actual intent of the real-world fixture. Adds a fullPage flag to fixturePages (default true); prob-matrix sets fullPage: false and the loop test.skip()s the full-page test. Deletes the orphan prob-matrix.png baselines (desktop + mobile). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent a05dede commit b2cc1b4

129 files changed

Lines changed: 339 additions & 217 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 31 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,45 @@
1-
name: Build Example Project [using jupyter-book]
1+
name: Build Fixtures Site for Visual Regression Tests
22
on:
33
push:
44
branches:
55
- main
66
pull_request:
7+
8+
# Visual regression tests run against the curated quantecon-book-theme-fixtures
9+
# site, pinned to a specific commit so the input doesn't move between theme
10+
# PRs. To bump the pin, update FIXTURES_SHA below.
11+
env:
12+
FIXTURES_REPO: QuantEcon/quantecon-book-theme-fixtures
13+
FIXTURES_SHA: d8ffc17c753ecf45fa25c6062827e1aa9de201b3
14+
715
jobs:
816
tests:
917
runs-on: ubuntu-latest
1018
steps:
11-
- name: Checkout
19+
- name: Checkout theme
20+
uses: actions/checkout@v6
21+
22+
- name: Checkout fixtures
1223
uses: actions/checkout@v6
13-
- name: Fetch lecture-python-programming
14-
shell: bash -l {0}
15-
run: |
16-
git clone --branch quantecon-book-theme https://github.com/QuantEcon/lecture-python-programming
17-
- name: Setup Anaconda
18-
uses: conda-incubator/setup-miniconda@v4
1924
with:
20-
auto-update-conda: true
21-
auto-activate-base: true
22-
miniconda-version: 'latest'
25+
repository: ${{ env.FIXTURES_REPO }}
26+
ref: ${{ env.FIXTURES_SHA }}
27+
path: fixtures
28+
29+
- name: Setup Python
30+
uses: actions/setup-python@v6
31+
with:
2332
python-version: "3.13"
24-
environment-file: lecture-python-programming/environment.yml
25-
activate-environment: lecture-python-programming
26-
- name: Install quantecon-book-theme
27-
shell: bash -l {0}
28-
run: |
29-
python -m pip install .
30-
- name: Display Conda Environment Versions
31-
shell: bash -l {0}
32-
run: conda list
33-
- name: Display Pip Versions
34-
shell: bash -l {0}
35-
run: pip list
36-
- name: Build HTML
37-
shell: bash -l {0}
33+
cache: "pip"
34+
35+
- name: Install fixtures build deps + this PR's theme
3836
run: |
39-
cd lecture-python-programming
40-
jb build lectures --path-output ./
37+
pip install -r fixtures/requirements.txt
38+
pip install .
39+
40+
- name: Build fixtures site
41+
working-directory: fixtures
42+
run: jb build . --warningiserror
4143

4244
# Visual Regression Testing
4345
- name: Setup Node.js
@@ -54,7 +56,7 @@ jobs:
5456
run: npm run test:visual
5557
continue-on-error: true
5658
env:
57-
SITE_PATH: lecture-python-programming/_build/html
59+
SITE_PATH: fixtures/_build/html
5860
- name: Upload Playwright Report
5961
uses: actions/upload-artifact@v7
6062
if: always()
@@ -83,7 +85,7 @@ jobs:
8385
- name: Preview Deploy to Netlify
8486
uses: nwtgck/actions-netlify@v3.0
8587
with:
86-
publish-dir: 'lecture-python-programming/_build/html/'
88+
publish-dir: 'fixtures/_build/html/'
8789
production-branch: main
8890
github-token: ${{ secrets.GITHUB_TOKEN }}
8991
deploy-message: "Preview Deploy from GitHub Actions"

.github/workflows/update-snapshots.yml

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,7 @@ on:
44
types: [created]
55

66
# Snapshots are generated against quantecon-book-theme-fixtures at this
7-
# pinned SHA. ci.yml will be migrated to the same fixtures repo in #390
8-
# (it still builds lecture-python-programming on main today); once that
9-
# lands, keep FIXTURES_SHA in sync between the two workflows.
7+
# pinned SHA. Keep FIXTURES_SHA in sync with .github/workflows/ci.yml.
108
env:
119
FIXTURES_REPO: QuantEcon/quantecon-book-theme-fixtures
1210
FIXTURES_SHA: d8ffc17c753ecf45fa25c6062827e1aa9de201b3

.gitignore

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -144,8 +144,9 @@ node_modules
144144
playwright-report/
145145
test-results/
146146

147-
# Visual test lecture site (cloned by tox -e visual)
148-
lecture-python-programming/
147+
# Visual test fixtures site (cloned by tox -e visual, or symlinked locally)
148+
fixtures
149+
fixtures/
149150

150151
# macOS visual test snapshots (tox uses macos/ dir, CI uses __snapshots__/)
151152
tests/visual/macos/

docs/developer/visual-testing.md

Lines changed: 61 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,16 @@
11
# Visual Regression Testing
22

3-
Playwright-based tests capture screenshots of key pages from a real lecture
4-
site build and compare them against baseline snapshots to detect unintended
5-
styling changes.
3+
Playwright-based tests capture screenshots of pages from the
4+
[`quantecon-book-theme-fixtures`](https://github.com/QuantEcon/quantecon-book-theme-fixtures)
5+
site and compare them against baseline snapshots to detect unintended
6+
styling changes. The fixtures repo is a curated, stable rendering target:
7+
the landing page, 12 synthetic pages (one per theme surface), and
8+
real-world lecture captures that previously exposed theme bugs.
9+
10+
The fixtures repo is **pinned to a specific commit** in the theme's CI
11+
workflows (`FIXTURES_SHA` in `.github/workflows/ci.yml` and
12+
`.github/workflows/update-snapshots.yml`), so the test input doesn't move
13+
between theme PRs.
614

715
```{contents}
816
:local:
@@ -18,10 +26,18 @@ $ tox -e visual
1826
```
1927

2028
This will:
21-
1. Clone `lecture-python-programming` (if not present)
22-
2. Build the lecture site with Jupyter Book
23-
3. Install Playwright and Chromium
24-
4. Run visual regression tests
29+
1. Clone `quantecon-book-theme-fixtures` (if not present)
30+
2. Check out the requested ref — defaults to `main`, overridable via the
31+
`FIXTURES_REF` env var (accepts SHA, branch, or tag)
32+
3. Build the fixtures site with `jb build .`
33+
4. Install Playwright and Chromium
34+
5. Run visual regression tests against the build
35+
36+
To pin to the same fixtures commit as CI for local testing:
37+
38+
```console
39+
$ FIXTURES_REF=<sha-from-ci.yml> tox -e visual
40+
```
2541

2642
### Updating Local Baselines
2743

@@ -30,11 +46,20 @@ $ tox -e visual-update
3046
```
3147

3248
Local baselines are stored in `tests/visual/macos/` (gitignored) and are
33-
separate from CI baselines in `tests/visual/__snapshots__/`.
49+
separate from CI baselines in `tests/visual/__snapshots__/`, since
50+
screenshot rendering differs between macOS and Linux.
3451

3552
## CI Integration
3653

37-
Visual tests run as part of the `ci.yml` workflow on every push and PR.
54+
Visual tests run as part of the `ci.yml` workflow on every push and PR:
55+
56+
1. Checks out the fixtures repo at `FIXTURES_SHA`
57+
2. Installs build deps via `pip install -r fixtures/requirements.txt`
58+
3. Installs the PR's theme via `pip install .`
59+
4. Builds the fixtures site with `jb build . --warningiserror`
60+
5. Runs Playwright against the build
61+
6. Deploys the fixtures site to Netlify as a PR preview (reviewers can
62+
click through every fixture page with this PR's theme applied)
3863

3964
### Updating CI Baselines via PR Comments
4065

@@ -46,14 +71,36 @@ Two commands are available as PR comments:
4671
| `/update-new-snapshots` | Adds only **missing** baselines — use for new tests |
4772

4873
Both commands:
49-
- Build the lecture site with the PR's theme changes
74+
- Check out the fixtures repo at `FIXTURES_SHA`
75+
- Build the fixtures site with the PR's theme applied
5076
- Run Playwright on Ubuntu for platform-consistent snapshots
5177
- Commit updated snapshots to the PR branch
5278
- Post a summary comment on the PR
5379

5480
The `/update-snapshots` command also uploads a `snapshot-update-diff` artifact
5581
with before/after images for review.
5682

83+
```{note}
84+
`issue_comment` workflows always load the workflow file from the **default
85+
branch**, not from the PR branch. If a PR changes `update-snapshots.yml`
86+
itself, the new workflow takes effect only after that change has been
87+
merged to `main`.
88+
```
89+
90+
## Bumping the fixtures pin
91+
92+
To consume new fixtures (e.g. after a real-world capture is added to the
93+
fixtures repo):
94+
95+
1. Open a PR on
96+
[`quantecon-book-theme-fixtures`](https://github.com/QuantEcon/quantecon-book-theme-fixtures)
97+
adding the page. See its `real-world/README.md` for capture conventions.
98+
2. Once merged, open a follow-up PR on this repo that bumps `FIXTURES_SHA`
99+
in both workflow files and (if needed) adds a corresponding test entry
100+
in `theme.spec.ts`.
101+
3. Comment `/update-new-snapshots` on that PR to seed baselines for any
102+
newly tested pages.
103+
57104
## Test Structure
58105

59106
```
@@ -67,5 +114,7 @@ tests/visual/
67114
## Configuration
68115

69116
Visual test configuration is in `playwright.config.ts` at the project root.
70-
The `tox.ini` `[testenv:visual]` section contains the full environment setup
71-
including dependencies like `jupyter-book`, `sphinx-exercise`, etc.
117+
The `tox.ini` `[testenv:visual]` and `[testenv:visual-update]` sections
118+
contain the full environment setup including the fixtures-repo clone +
119+
checkout dance and dependencies like `jupyter-book`, `sphinx-exercise`,
120+
etc.

playwright.config.ts

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,11 @@ import { defineConfig, devices } from "@playwright/test";
33
/**
44
* Playwright configuration for visual regression testing.
55
*
6-
* Tests are run against a locally served build of lecture-python-programming
7-
* to verify theme styling hasn't regressed.
6+
* Tests are run against a locally served build of the
7+
* quantecon-book-theme-fixtures site to verify theme styling hasn't
8+
* regressed. The fixtures repo is checked out as a sibling directory
9+
* (`fixtures/`) by CI; local users can clone it manually or use
10+
* `tox -e visual`, which clones it automatically.
811
*/
912
export default defineConfig({
1013
testDir: "./tests/visual",
@@ -61,10 +64,11 @@ export default defineConfig({
6164
? `{testDir}/{testFileDir}/${process.env.SNAPSHOT_DIR}/{projectName}/{arg}{ext}`
6265
: "{testDir}/{testFileDir}/__snapshots__/{projectName}/{arg}{ext}",
6366

64-
// Web server to serve the built lecture site
65-
// Path varies: CI uses _build/html, tox uses lectures/_build/html
67+
// Web server to serve the built fixtures site.
68+
// CI sets SITE_PATH=fixtures/_build/html; local default matches the tox
69+
// clone path.
6670
webServer: {
67-
command: `python -m http.server 8000 --directory ${process.env.SITE_PATH || "lecture-python-programming/lectures/_build/html"}`,
71+
command: `python -m http.server 8000 --directory ${process.env.SITE_PATH || "fixtures/_build/html"}`,
6872
url: "http://localhost:8000",
6973
reuseExistingServer: !process.env.CI,
7074
timeout: 120 * 1000,

0 commit comments

Comments
 (0)