Skip to content

Commit 44b75d7

Browse files
OnibenjoOnibenjo
authored
chore: gate Percy snapshots to visual-affecting changes (#2580)
1 parent a483660 commit 44b75d7

6 files changed

Lines changed: 249 additions & 8 deletions

File tree

‎.github/PERCY.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Percy snapshots
2+
3+
This repo runs visual regression tests via [Percy](https://percy.io/). Three workflows orchestrate it:
4+
5+
- `.github/workflows/percy-pr.yaml` — PRs from this repo targeting `main`
6+
- `.github/workflows/percy-fork-pr.yaml` — PRs from forks targeting `main` (uses `pull_request_target` with an approval gate)
7+
- `.github/workflows/percy-baseline.yaml` — pushes to `main` (refreshes the baseline that PRs compare against)
8+
9+
The shared gate logic lives in `.github/actions/percy-gate/action.yml`.
10+
11+
## When Percy runs
12+
13+
For PRs (both internal and fork) the `decide` job evaluates, in order:
14+
15+
1. **`run-percy` label present on the PR?** → RUN.
16+
2. **Author is `dependabot[bot]` or `renovate[bot]`?** → SKIP (bots must use the label to opt in).
17+
3. **Diff touches a watched path?** → RUN.
18+
4. **Otherwise** → SKIP.
19+
20+
For pushes to `main` (baseline):
21+
22+
1. **Workflow manually dispatched?** → RUN.
23+
2. **Push diff touches a watched path?** → RUN.
24+
3. **Otherwise** → SKIP.
25+
26+
When the gate decides to skip, the `snapshot` job is skipped but the workflow exits successfully — branch protection (which treats `skipped` as `success` for required checks) is unaffected.
27+
28+
## Watched paths
29+
30+
Defined once in `.github/actions/percy-gate/action.yml`:
31+
32+
- `static/sass/**`
33+
- `static/js/**`
34+
- `templates/**`
35+
- `navigation.yaml`
36+
- `secondary-navigation.yaml`
37+
- `snapshots.js`
38+
- `test-links.yaml`
39+
- `package.json`
40+
- `yarn.lock`
41+
42+
To extend the list, edit the `filters:` block in `.github/actions/percy-gate/action.yml`. The change applies to all three workflows automatically.
43+
44+
## How to force a Percy run
45+
46+
### On a PR
47+
48+
Add the `run-percy` label. Adding the label fires a `labeled` event, the gate re-evaluates, and the snapshot job runs.
49+
50+
Reach for this when:
51+
52+
- The PR changes visuals via a path we don't watch (e.g. a Python view that swaps template variables).
53+
- A Renovate or Dependabot PR bumps a visual-affecting dependency (e.g. `vanilla-framework`) and you want a snapshot.
54+
- You want a one-off sanity-check snapshot on any PR.
55+
56+
### On `main` (refresh the baseline manually)
57+
58+
Go to **Actions → Update Percy Baseline → Run workflow**, select `main`, click Run.
59+
60+
Reach for this when:
61+
62+
- A merged PR's change wasn't caught by the path filter and visual drift later surfaces in production.
63+
- You want a fresh baseline before a release.
64+
65+
## Rapid pushes are de-duplicated
66+
67+
PR workflows declare a `concurrency` group with `cancel-in-progress: true`, so pushing several commits in quick succession only completes the latest run — earlier in-flight Percy runs for the same PR are cancelled. The baseline workflow does not cancel itself (every `main` push completes its baseline independently).
68+
69+
## Troubleshooting
70+
71+
**"Percy didn't run on my PR with a CSS change."**
72+
Check the watched-paths list above. If the path is genuinely visual-affecting and not covered, add it to `.github/actions/percy-gate/action.yml`. For a one-off run, add the `run-percy` label.
73+
74+
**"The `Take Percy snapshots` check shows as skipped."**
75+
Expected when the gate decided to skip. The `Decide whether to run Percy` job ran to completion and reported success — branch protection is satisfied.
76+
77+
**"Percy ran on a PR that didn't change visuals."**
78+
Check whether the diff touches any watched path — `package.json` and `yarn.lock` will match any dependency bump. Bots are blocked by default, but a human PR touching these triggers a run.

‎.github/actions/percy-gate/action.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: Percy Gate
2+
description: >
3+
Decides whether a Percy snapshot run should proceed. Combines a path filter
4+
with a label override and a bot-author skip for PR contexts, and a
5+
workflow_dispatch override for baseline (push) contexts.
6+
7+
inputs:
8+
mode:
9+
description: >
10+
'pr' for PR workflows (checks label, bot author, then paths). 'baseline'
11+
for the main-push workflow (checks workflow_dispatch, then paths).
12+
required: true
13+
14+
outputs:
15+
should_run:
16+
description: 'true if the calling workflow should proceed to take snapshots'
17+
value: ${{ steps.evaluate.outputs.should_run }}
18+
19+
runs:
20+
using: composite
21+
steps:
22+
- name: Filter changed paths
23+
id: filter
24+
uses: dorny/paths-filter@d1c1ffe0248fe513906c8e24db8ea791d46f8590 # v3.0.3
25+
with:
26+
filters: |
27+
visual:
28+
- 'static/sass/**'
29+
- 'static/js/**'
30+
- 'templates/**'
31+
- 'navigation.yaml'
32+
- 'secondary-navigation.yaml'
33+
- 'snapshots.js'
34+
- 'test-links.yaml'
35+
- 'package.json'
36+
- 'yarn.lock'
37+
38+
- name: Evaluate gate
39+
id: evaluate
40+
shell: bash
41+
env:
42+
MODE: ${{ inputs.mode }}
43+
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-percy') }}
44+
ACTOR: ${{ github.event.pull_request.user.login }}
45+
VISUAL_CHANGED: ${{ steps.filter.outputs.visual }}
46+
DISPATCHED: ${{ github.event_name == 'workflow_dispatch' }}
47+
run: |
48+
set -e
49+
if [ "$MODE" = "baseline" ]; then
50+
if [ "$DISPATCHED" = "true" ]; then
51+
echo "RUN: manually dispatched"
52+
echo "should_run=true" >> "$GITHUB_OUTPUT"
53+
exit 0
54+
fi
55+
if [ "$VISUAL_CHANGED" = "true" ]; then
56+
echo "RUN: visual-affecting files changed on this push"
57+
echo "should_run=true" >> "$GITHUB_OUTPUT"
58+
exit 0
59+
fi
60+
echo "SKIP: no visual-affecting changes on this push"
61+
echo "should_run=false" >> "$GITHUB_OUTPUT"
62+
exit 0
63+
fi
64+
if [ "$HAS_LABEL" = "true" ]; then
65+
echo "RUN: run-percy label present"
66+
echo "should_run=true" >> "$GITHUB_OUTPUT"
67+
exit 0
68+
fi
69+
if [ "$ACTOR" = "dependabot[bot]" ] || [ "$ACTOR" = "renovate[bot]" ]; then
70+
echo "SKIP: bot author ($ACTOR) without run-percy label"
71+
echo "should_run=false" >> "$GITHUB_OUTPUT"
72+
exit 0
73+
fi
74+
if [ "$VISUAL_CHANGED" = "true" ]; then
75+
echo "RUN: visual-affecting files changed"
76+
echo "should_run=true" >> "$GITHUB_OUTPUT"
77+
exit 0
78+
fi
79+
echo "SKIP: no visual-affecting changes and no run-percy label"
80+
echo "should_run=false" >> "$GITHUB_OUTPUT"

‎.github/workflows/percy-baseline.yaml‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,41 @@
1-
# This workflow updates the Percy baseline on every push to main.
2-
# This allows pull requests to be compared against baseline test results.
1+
# Updates the Percy baseline on pushes to main that touch visual-affecting files.
2+
# See .github/PERCY.md for when this runs, the watched-paths list, and how to refresh manually.
33

44
name: Update Percy Baseline
55

66
on:
77
push:
88
branches:
99
- main
10+
workflow_dispatch:
1011

1112
jobs:
13+
decide:
14+
name: Decide whether to update baseline
15+
runs-on: ubuntu-latest
16+
permissions:
17+
contents: read
18+
outputs:
19+
should_run: ${{ steps.gate.outputs.should_run }}
20+
steps:
21+
- uses: actions/checkout@v4
22+
with:
23+
sparse-checkout: .github/actions/percy-gate
24+
sparse-checkout-cone-mode: false
25+
persist-credentials: false
26+
27+
- id: gate
28+
uses: ./.github/actions/percy-gate
29+
with:
30+
mode: baseline
31+
1232
snapshot:
1333
name: Take Percy snapshots
1434
runs-on: ubuntu-latest
35+
needs: decide
36+
if: needs.decide.result == 'success' && needs.decide.outputs.should_run == 'true'
37+
permissions:
38+
contents: read
1539
steps:
1640
- name: Checkout SCM
1741
uses: actions/checkout@v4
@@ -20,4 +44,4 @@ jobs:
2044
with:
2145
branch_name: main
2246
commitsh: ${{ github.sha }}
23-
percy_token_write: ${{ secrets.PERCY_TOKEN_WRITE }}
47+
percy_token_write: ${{ secrets.PERCY_TOKEN_WRITE }}

‎.github/workflows/percy-fork-pr.yaml‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# See .github/PERCY.md for when Percy runs, the watched-paths list, and how to force a run.
2+
13
name: Percy testing on PRs from forks
24
run-name: "Testing Percy on fork PR #${{ github.event.pull_request.number }}"
35
# This workflow handles Percy testing for PRs from forks securely
@@ -12,18 +14,42 @@ on:
1214
- synchronize
1315
- reopened
1416
- ready_for_review
17+
- labeled
18+
- unlabeled
1519

1620
permissions:
1721
contents: read
1822
pull-requests: read
1923

24+
concurrency:
25+
group: percy-fork-${{ github.event.pull_request.number }}
26+
cancel-in-progress: true
27+
2028
jobs:
29+
decide:
30+
name: Decide whether to run Percy
31+
runs-on: ubuntu-latest
32+
# Only run for forks; internal PRs use percy-pr.yaml
33+
if: github.event.pull_request.head.repo.full_name != github.repository
34+
outputs:
35+
should_run: ${{ steps.gate.outputs.should_run }}
36+
steps:
37+
- uses: actions/checkout@v4
38+
with:
39+
sparse-checkout: .github/actions/percy-gate
40+
sparse-checkout-cone-mode: false
41+
persist-credentials: false
42+
43+
- id: gate
44+
uses: ./.github/actions/percy-gate
45+
with:
46+
mode: pr
47+
2148
snapshot:
2249
name: Take Percy snapshots
2350
runs-on: ubuntu-latest
24-
needs: security-check
25-
# Ensure we only run for forks (internal PRs use a different workflow)
26-
if: github.event.pull_request.head.repo.full_name != github.repository
51+
needs: decide
52+
if: needs.decide.result == 'success' && needs.decide.outputs.should_run == 'true'
2753

2854
# The environment "percy-testing" handles the approval gate
2955
environment: percy-testing

‎.github/workflows/percy-pr.yaml‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
# See .github/PERCY.md for when Percy runs, the watched-paths list, and how to force a run.
12

23
name: Percy testing on PRs from same repo
34
run-name: Testing Percy on branch ${{ github.head_ref }}
@@ -12,12 +13,42 @@ on:
1213
- synchronize
1314
- reopened
1415
- ready_for_review
16+
- labeled
17+
- unlabeled
18+
19+
concurrency:
20+
group: percy-${{ github.event.pull_request.number }}
21+
cancel-in-progress: true
1522

1623
jobs:
24+
decide:
25+
name: Decide whether to run Percy
26+
runs-on: ubuntu-latest
27+
if: github.event.pull_request.head.repo.full_name == github.repository
28+
permissions:
29+
contents: read
30+
pull-requests: read
31+
outputs:
32+
should_run: ${{ steps.gate.outputs.should_run }}
33+
steps:
34+
- uses: actions/checkout@v4
35+
with:
36+
sparse-checkout: .github/actions/percy-gate
37+
sparse-checkout-cone-mode: false
38+
persist-credentials: false
39+
40+
- id: gate
41+
uses: ./.github/actions/percy-gate
42+
with:
43+
mode: pr
44+
1745
snapshot:
1846
name: Take Percy snapshots
1947
runs-on: ubuntu-latest
20-
if: github.event.pull_request.head.repo.full_name == github.repository
48+
needs: decide
49+
if: needs.decide.result == 'success' && needs.decide.outputs.should_run == 'true'
50+
permissions:
51+
contents: read
2152
steps:
2253
- name: Checkout SCM
2354
uses: actions/checkout@v4
@@ -26,4 +57,4 @@ jobs:
2657
with:
2758
branch_name: ${{ github.head_ref }}
2859
commitsh: ${{ github.sha }}
29-
percy_token_write: ${{ secrets.PERCY_TOKEN_WRITE }}
60+
percy_token_write: ${{ secrets.PERCY_TOKEN_WRITE }}

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ PERCY_BROWSER_EXECUTABLE=/Applications/Chromium.app/Contents/MacOS/Chromium
3232
PERCY_POSTINSTALL_BROWSER=false
3333
```
3434

35+
In CI, Percy snapshots only run when a PR (or push to `main`) touches files that can affect rendered output (SCSS, JS, templates, nav YAMLs, snapshot config, `package.json`/`yarn.lock`). See [.github/PERCY.md](.github/PERCY.md) for the full behaviour, watched-paths list, and how to force a run via the `run-percy` label.
36+
3537
## Environment variables
3638

3739
Environment variables are read from the available shell. For the charm, these are prepended with the prefix `FLASK_`, which we strip before re-inserting them into the environment.

0 commit comments

Comments
 (0)