You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Migrate visual regression tests to quantecon-book-theme-fixtures (#390)
* Migrate visual regression tests to quantecon-book-theme-fixtures
Replace the lecture-python-programming clone (moving target, conda build,
~3-5 min per CI run) with the curated quantecon-book-theme-fixtures repo
pinned at SHA d8ffc17. The fixtures site builds in ~30-60s with pip-only
installs, has stable inputs across PRs, and covers more theme surfaces
(definition lists, deeply nested TOC, etc.) than the previous lecture
subset could.
Changes:
- ci.yml + update-snapshots.yml: checkout fixtures repo at FIXTURES_SHA,
pip-install requirements + this PR's theme, jb build --warningiserror,
point Playwright at fixtures/_build/html
- tox.ini: visual + visual-update envs clone the fixtures repo (FIXTURES_REF
env var lets you pin a SHA locally)
- playwright.config.ts: default SITE_PATH points at fixtures/_build/html
- theme.spec.ts: rewrite page list for 13 synthetic + 1 real-world fixture;
add definition-list tests (previously skipped — no <dl> in the lecture
site); drop hasPlots distinction (fixtures don't execute notebooks)
- tests/visual/README.md: rewritten around fixtures workflow
- .gitignore: replace lecture-python-programming entry with fixtures
- Delete stale ubuntu snapshots under tests/visual/__snapshots__/ —
page paths changed; comment /update-new-snapshots on this PR to seed
fresh baselines
The /update-new-snapshots and /update-snapshots PR comment commands still
work the same way; they now build the fixtures site instead of the lecture
site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Address Copilot review feedback on #390
- tox.ini: clone fixtures without --branch, then fetch + checkout
${FIXTURES_REF:-main} on every run. Previously --branch only accepted
branch/tag names (not SHAs, despite the README documenting SHA usage)
and the [ -d fixtures ] guard meant FIXTURES_REF changes were silently
ignored on subsequent runs.
- tox.ini: always run jb build (sphinx is incremental). The previous
skip-if-built guard caused tox -e visual to serve stale theme assets
when the theme changed between runs — CI was unaffected but local
dev could silently test against the old theme.
- theme.spec.ts: gate the loose maxDiffPixelRatio: 0.05 on a per-page
hasMath flag. Non-math pages now keep the strict default tolerance
from playwright.config.ts so sub-pixel regressions still get caught.
- tests/visual/README.md: clarify that the test set is the landing
page + 12 synthetic pages + real-world captures (was: "12 synthetic
pages" alone).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update visual-testing.md for fixtures-based workflow
Missed in the initial migration commit — the user-facing developer docs at
docs/developer/visual-testing.md still described the lecture-python-programming
flow (clone + jb build lectures + …). Rewritten to match the rest of the PR:
fixtures clone, FIXTURES_SHA pinning, FIXTURES_REF for local pins, and the
issue_comment + default-branch gotcha that motivated the #392 split.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* UPDATE: Visual regression snapshots (new)
* Bump toHaveScreenshot timeout to 30s for long-page fixtures
The /update-new-snapshots run on #390 successfully generated baselines for
13 of 14 fixture pages, but prob-matrix full-page screenshots timed out
at Playwright's default 5000ms on both desktop and mobile — prob_matrix.md
is ~1900 source lines, much taller than the other fixtures, and Playwright
couldn't stitch the full-page capture in time. Header and sidebar
baselines for prob-matrix were generated fine.
Bumping the toHaveScreenshot timeout to 30s gives long-page captures
plenty of headroom while leaving short captures unaffected (they still
finish in well under a second).
After this lands, re-comment /update-new-snapshots on this PR to seed the
two missing prob-matrix baselines (workflow leaves existing baselines
alone, only creates missing ones).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Move screenshot timeout to per-assertion call (real fix)
The previous attempt put timeout: 30000 in expect.toHaveScreenshot in
playwright.config.ts, but timeout isn't a valid key in that config block
(Playwright silently ignores it — the latest /update-new-snapshots run
still hit "Timeout 5000ms exceeded" on prob-matrix). The valid places
are expect.timeout (too broad — affects all matchers) or per-assertion.
Adding timeout: 30000 directly to the full-page toHaveScreenshot call in
the spec, which is the canonical Playwright pattern. Reverting the
no-op config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* UPDATE: Visual regression snapshots (new)
* ci: trigger run on latest branch state
The previous CI run was on c6d3a60 (the timeout fix), before the bot's
snapshot commit 4d48a02 landed. Commits made by GITHUB_TOKEN don't
trigger downstream workflows by design, so the baselines committed by
the /update-new-snapshots bot never got a CI pass. This empty commit
forces a fresh run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Disable full-page screenshot test for prob-matrix fixture
The CI run after the bot's baseline commit revealed that the prob-matrix
full-page capture isn't stable: expected height 41452px, captured height
40693px on subsequent runs (759px difference). Playwright requires
identical image dimensions for comparison, so no tolerance setting can
help. The variance comes from MathJax font-metric resolution + image
loading + table layout — networkidle + MathJax.startup.promise resolves
before the page actually stops growing.
Real-world fixtures inherently have this characteristic; the synthetic
math.html page already provides full-page math regression coverage with
stable dimensions. For prob-matrix, header + sidebar region tests still
run (those are stable theme elements) and the Netlify preview still
renders the full page for human visual review — which is the actual
intent of the real-world fixture.
Adds a fullPage flag to fixturePages (default true); prob-matrix sets
fullPage: false and the loop test.skip()s the full-page test. Deletes
the orphan prob-matrix.png baselines (desktop + mobile).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
0 commit comments