Skip to content

Commit f6fbd92

Browse files
jaggujijagguji
committed
Add real-world benchmark gate + fixture guard
Benchmark gate (benchmark/ + .github/workflows/benchmark.yml): on every PR a manual 'Approve and run' button (environment-gated) runs the PR's generator against the pinned packages in benchmark/packages.json, compiles the output with ReScript 12, diffs it against committed baselines, and compares quality buckets — FAIL on any regression, WARN on equal-or-better diffs (accept via npm run bench:update), sticky PR comment with the verdict table. Replaces the unpinned, non-blocking blend integration job. Fixture guard in ci.yml (closes #14): a PR touching src/extract|emit|resolve.mjs must also touch test/golden/cases/ or docs/TYPE_MAPPING.md (label no-fixture-needed opts out). Supporting changes: --json-summary flag on the CLI (machine-readable bucket counts), shared snapshot-diff helpers extracted to test/lib/diff.mjs, bench / bench:update npm scripts. benchmark/ is not published (files allowlist).
1 parent 70ccfe9 commit f6fbd92

235 files changed

Lines changed: 29073 additions & 68 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/benchmark.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Real-world benchmark gate — runs the PR's generator against the pinned packages in
2+
# benchmark/packages.json and compares with committed baselines (see benchmark/README.md).
3+
#
4+
# ONE-BUTTON FLOW: the job is gated by the `benchmark` GitHub environment (Settings →
5+
# Environments → `benchmark` → required reviewer = maintainer). Every PR run parks in
6+
# "Waiting"; the PR page shows "Review pending deployments" → **Approve and run**.
7+
# Heavy work never starts without that click.
8+
name: Benchmark
9+
10+
on:
11+
pull_request:
12+
workflow_dispatch: # manual re-runs / debugging from the Actions tab
13+
14+
concurrency:
15+
group: benchmark-${{ github.ref }}
16+
cancel-in-progress: true # a new push supersedes any pending approval / running benchmark
17+
18+
jobs:
19+
benchmark:
20+
name: Benchmark (pinned real packages)
21+
runs-on: ubuntu-latest
22+
environment: benchmark # ← the approval button; not a required check, normal CI gates merges
23+
permissions:
24+
contents: read
25+
pull-requests: write # sticky result comment (silently unavailable on fork PRs)
26+
steps:
27+
- uses: actions/checkout@v4
28+
29+
- name: Setup Node.js
30+
uses: actions/setup-node@v4
31+
with:
32+
node-version: 20
33+
cache: npm
34+
35+
- run: npm ci
36+
37+
# Sandbox node_modules are reused when the committed lockfiles are unchanged
38+
# (run.mjs stamps each sandbox and skips `npm ci` on a hash match).
39+
- name: Cache benchmark sandboxes
40+
uses: actions/cache@v4
41+
with:
42+
path: benchmark/.work/*/sandbox/node_modules
43+
key: bench-sandbox-${{ runner.os }}-${{ hashFiles('benchmark/packages.json', 'benchmark/sandbox-template/**', 'benchmark/baselines/*/package-lock.json') }}
44+
45+
- name: Run benchmark
46+
run: npm run bench
47+
48+
- name: Job summary
49+
if: always()
50+
run: cat benchmark/.work/results.md >> "$GITHUB_STEP_SUMMARY" || echo "no results.md produced" >> "$GITHUB_STEP_SUMMARY"
51+
52+
- name: Sticky PR comment
53+
if: always() && github.event_name == 'pull_request'
54+
uses: marocchino/sticky-pull-request-comment@v2
55+
continue-on-error: true # read-only token on fork PRs
56+
with:
57+
header: bindgen-benchmark
58+
path: benchmark/.work/results.md

‎.github/workflows/ci.yml‎

Lines changed: 18 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -80,30 +80,27 @@ jobs:
8080
- name: Compile goldens
8181
run: npm run test:compile
8282

83-
# Integration: generate bindings for the latest published blend and compile them.
84-
# Informational — depends on npm + the published package, so it must not block CI.
85-
integration:
86-
name: Generate + compile (blend latest)
83+
# Maintenance-loop guard (issue #14): a mapping change is not done until its golden
84+
# fixture and TYPE_MAPPING.md row land in the SAME PR (see CLAUDE.md). Pure refactors
85+
# can opt out with the `no-fixture-needed` label.
86+
fixture-guard:
87+
name: Fixture guard (new cases need test cases)
8788
runs-on: ubuntu-latest
88-
continue-on-error: true
89+
if: github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'no-fixture-needed')
8990
steps:
9091
- uses: actions/checkout@v4
91-
92-
- name: Setup Node.js
93-
uses: actions/setup-node@v4
9492
with:
95-
node-version: 20
96-
cache: npm
97-
98-
- run: npm ci
99-
100-
- name: Install ReScript sandbox deps
101-
run: npm --prefix test/sandbox ci || npm --prefix test/sandbox install
102-
103-
- name: Generate blend bindings
104-
run: npm run gen -- --pkg @juspay/blend-design-system@latest --out test/sandbox/src --from "@juspay/blend-design-system" --webapi --yes
93+
fetch-depth: 0
10594

106-
- name: Compile-check generated bindings
95+
- name: Mapping changes must ship with fixtures
10796
run: |
108-
printf 'module File = {\n type t\n}\nmodule FileList = {\n type t\n}\nmodule FormData = {\n type t\n}\n' > test/sandbox/src/Webapi.res
109-
npx --prefix test/sandbox rescript build || (cd test/sandbox && npx rescript build)
97+
BASE="origin/${{ github.event.pull_request.base.ref }}"
98+
CHANGED=$(git diff --name-only "$BASE"...HEAD)
99+
echo "$CHANGED"
100+
if echo "$CHANGED" | grep -qE '^src/(extract|emit|resolve)\.mjs$' \
101+
&& ! echo "$CHANGED" | grep -qE '^(test/golden/cases/|docs/TYPE_MAPPING\.md)'; then
102+
echo '::error::This PR changes the mapping pipeline (src/extract|emit|resolve.mjs) but adds/updates no golden case and no docs/TYPE_MAPPING.md row.'
103+
echo '::error::Per CLAUDE.md, a mapping change is not done until all three land together: TYPE_MAPPING.md row + fixture under test/golden/cases/ + regenerated golden (npm run test:golden:update). Pure refactors: add the `no-fixture-needed` label.'
104+
exit 1
105+
fi
106+
echo "✅ fixture guard passed"

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,6 @@ test/binding-compare/Card.*.res
2727

2828
# scratch manifest from sandbox generation runs
2929
test/sandbox/src/.bindgen-manifest.json
30+
31+
# benchmark runtime scratch (sandboxes, logs, results) — baselines/ ARE committed
32+
benchmark/.work/

‎CLAUDE.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,16 @@ A new or changed mapping is not done until all three are updated together:
4040
| `npm run test:golden:update` | regenerate goldens (after an intentional change) |
4141
| `npm run test:compile` | compile every golden with ReScript in `test/sandbox` (needs sandbox deps) |
4242
| `npm run gen -- --pkg <name> --out <dir> --report` | generate bindings for a package |
43-
44-
CI (`.github/workflows/ci.yml`) runs the smoke+golden diff and the compile check on every PR, both
45-
**blocking** — generated output cannot drift from `docs/TYPE_MAPPING.md` without a failing build.
43+
| `npm run bench` | real-world benchmark: run the checkout against the pinned packages in `benchmark/packages.json`, diff vs committed baselines (see `benchmark/README.md`) |
44+
| `npm run bench:update` | accept intentional output changes — regenerate `benchmark/baselines/` and commit the diff in the same PR |
45+
46+
CI (`.github/workflows/ci.yml`) runs the smoke+golden diff, the compile check, and a
47+
**fixture guard** (a PR touching `src/extract|emit|resolve.mjs` must also touch
48+
`test/golden/cases/` or `docs/TYPE_MAPPING.md`; label `no-fixture-needed` opts out) on every
49+
PR, all **blocking** — generated output cannot drift from `docs/TYPE_MAPPING.md` without a
50+
failing build. `.github/workflows/benchmark.yml` is the opt-in heavy gate: approve it from
51+
the PR's "Review pending deployments" button to run the real-package benchmark before a
52+
release.
4653

4754
## Conventions
4855

‎benchmark/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Benchmark gate — real-world validation before release
2+
3+
This suite runs the **current checkout's** generator against version-pinned, well-known
4+
npm packages (`packages.json`) and compares the result with committed baselines. It is
5+
the pre-release safety net for the blend pipeline: a green run means this version of
6+
the tool can safely regenerate `@juspay/rescript-blend`'s bindings.
7+
8+
It does **not** ship to npm — `package.json`'s `files` allowlist excludes `benchmark/`
9+
(like `test/`).
10+
11+
## The one-click button on PRs
12+
13+
`.github/workflows/benchmark.yml` triggers on every PR but the job is gated by the
14+
`benchmark` GitHub **environment**, so it parks in *Waiting* until approved:
15+
16+
1. Open the PR → checks area shows **"Review pending deployments"**.
17+
2. Click it → **Approve and run**. That's the button.
18+
3. The run posts a sticky comment on the PR with the per-package verdict table
19+
(also visible in the workflow job summary).
20+
21+
**One-time setup** (already done if the check works): repo *Settings → Environments →
22+
New environment* `benchmark` → add the maintainer under *Required reviewers*.
23+
24+
Fork-PR caveat: `GITHUB_TOKEN` is read-only for forks, so the sticky comment silently
25+
fails there — the job summary and the check status still work.
26+
27+
## Verdicts
28+
29+
| Verdict | Meaning |
30+
|---|---|
31+
| ✅ PASS | Output byte-identical to baseline (incl. an unchanged known-gap package) |
32+
| ⚠️ WARN | Output changed but compiles and quality is equal-or-better — likely an intentional improvement |
33+
| ❌ FAIL | Generator crashed, bindings stopped compiling, new warnings, or buckets regressed (broken↑ / usable↓ / review↑) |
34+
35+
The job exits 1 (red check) only on FAIL.
36+
37+
## Commands
38+
39+
| Command | What it does |
40+
|---|---|
41+
| `npm run bench` | verify all packages against baselines (what CI runs) |
42+
| `npm run bench:update` | regenerate `baselines/<slug>/{bindings/,metrics.json,package-lock.json}` |
43+
| `node benchmark/run.mjs --only react-markdown` | debug one package (slug or name) |
44+
45+
## Accepting an intentional output change (WARN → PASS)
46+
47+
```sh
48+
npm run bench:update
49+
git add benchmark/baselines
50+
# commit in the same PR — the binding changes become part of the reviewable diff
51+
```
52+
53+
For judging a WARN before accepting it, the AI-review `bindgen-probe` skill can be run
54+
locally on the changed package — it is deliberately **not** part of this CI gate (the
55+
gate stays deterministic; no LLM, no secrets).
56+
57+
## Adding a package
58+
59+
Add an entry to `packages.json` (exact `version`, optional `flags` like `--webapi`),
60+
then `npm run bench:update` and commit the new baseline. Determinism comes from the
61+
exact pin + the committed sandbox `package-lock.json` (pins the transitive `.d.ts`
62+
surface).

‎benchmark/baselines/_juspay_blend-design-system/bindings/Accordion.res‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
@module("@juspay/blend-design-system") @react.component
2+
external make: (
3+
~children: React.element,
4+
~accordionType: AccordionTypes.accordionType=?,
5+
~defaultValue: CommonTypes.stringOrStringArray=?,
6+
~value: CommonTypes.stringOrStringArray=?,
7+
~isMultiple: bool=?,
8+
~onValueChange: CommonTypes.stringOrStringArray => unit=?,
9+
) => React.element = "Accordion"

‎benchmark/baselines/_juspay_blend-design-system/bindings/AccordionItem.res‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
@module("@juspay/blend-design-system") @react.component
2+
external make: (
3+
~value: string,
4+
~title: string,
5+
~subtext: string=?,
6+
~leftSlot: React.element=?,
7+
~rightSlot: React.element=?,
8+
~subtextSlot: React.element=?,
9+
~triggerSlot: React.element=?,
10+
~triggerSlotWidth: CommonTypes.stringOrNumber=?,
11+
~children: React.element,
12+
~isDisabled: bool=?,
13+
~chevronPosition: AccordionTypes.accordionChevronPosition=?,
14+
~accordionType: AccordionTypes.accordionType=?,
15+
~isFirst: bool=?,
16+
~isLast: bool=?,
17+
~isIntermediate: bool=?,
18+
~currentValue: CommonTypes.stringOrStringArray=?,
19+
) => React.element = "AccordionItem"

‎benchmark/baselines/_juspay_blend-design-system/bindings/AccordionTypes.res‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
type accordionType =
2+
| @as("border") Border
3+
| @as("noBorder") NoBorder
4+
type accordionChevronPosition =
5+
| @as("left") Left
6+
| @as("right") Right
7+
type gapConfig = {
8+
border: string,
9+
noBorder: string,
10+
}
11+
type borderConfig4 = {
12+
disabled: string,
13+
default: string,
14+
hover: string,
15+
active: string,
16+
@as("open") open_: string,
17+
}
18+
type backgroundColorConfig5 = {
19+
border: borderConfig4,
20+
noBorder: borderConfig4,
21+
}
22+
type colorConfig4 = {
23+
disabled: string,
24+
default: string,
25+
hover: string,
26+
active: string,
27+
@as("open") open_: string,
28+
}
29+
type titleConfig2 = {
30+
fontSize: string,
31+
fontWeight: string,
32+
color: colorConfig4,
33+
}
34+
type subtextConfig = {
35+
fontSize: string,
36+
gap: string,
37+
color: colorConfig4,
38+
}
39+
type textConfig10 = {
40+
title: titleConfig2,
41+
subtext: subtextConfig,
42+
}
43+
type slotConfig2 = {
44+
maxWidth: string,
45+
}
46+
type triggerConfig2 = {
47+
backgroundColor: backgroundColorConfig5,
48+
border: backgroundColorConfig5,
49+
padding: gapConfig,
50+
text: textConfig10,
51+
slot?: slotConfig2,
52+
}
53+
type separatorConfig = {
54+
color: gapConfig,
55+
}
56+
type accordionTokenType = {
57+
gap: gapConfig,
58+
borderRadius: gapConfig,
59+
trigger: triggerConfig2,
60+
separator: separatorConfig,
61+
}
62+
type responsiveAccordionTokens = {
63+
sm: accordionTokenType,
64+
lg: accordionTokenType,
65+
}

‎benchmark/baselines/_juspay_blend-design-system/bindings/AccordionV2Types.res‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
type titleConfig7 = {
2+
fontSize: string,
3+
fontWeight: string,
4+
lineHeight: string,
5+
color: AccordionTypes.colorConfig4,
6+
}
7+
type subtextConfig3 = {
8+
fontSize: string,
9+
fontWeight: string,
10+
lineHeight: string,
11+
gap: string,
12+
color: AccordionTypes.colorConfig4,
13+
}
14+
type textConfig24 = {
15+
gap: string,
16+
title: titleConfig7,
17+
subtext: subtextConfig3,
18+
}
19+
type slotConfig6 = {
20+
height: string,
21+
}
22+
type triggerConfig7 = {
23+
content: ChartsTypes.slotsConfig,
24+
backgroundColor: AccordionTypes.backgroundColorConfig5,
25+
border: AccordionTypes.backgroundColorConfig5,
26+
padding: AccordionTypes.gapConfig,
27+
text: textConfig24,
28+
slot: slotConfig6,
29+
}
30+
type chevronConfig2 = {
31+
height: string,
32+
color: AccordionTypes.borderConfig4,
33+
}
34+
type accordionV2TokensType = {
35+
gap: AccordionTypes.gapConfig,
36+
borderRadius: AccordionTypes.gapConfig,
37+
trigger: triggerConfig7,
38+
separator: AccordionTypes.separatorConfig,
39+
chevron: chevronConfig2,
40+
}
41+
type responsiveAccordionV2Tokens = {
42+
sm: accordionV2TokensType,
43+
lg: accordionV2TokensType,
44+
}

‎benchmark/baselines/_juspay_blend-design-system/bindings/Alert.res‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
@module("@juspay/blend-design-system") @react.component
2+
external make: (
3+
~heading: string,
4+
~description: string,
5+
~variant: AlertTypes.alertVariant=?,
6+
~style: AlertTypes.alertStyle=?,
7+
~primaryAction: AlertTypes.alertAction=?,
8+
~secondaryAction: AlertTypes.alertAction=?,
9+
~onClose: unit => unit=?,
10+
~icon: React.element=?,
11+
~actionPlacement: AlertTypes.alertActionPlacement=?,
12+
~maxWidth: string=?,
13+
~minWidth: string=?,
14+
~width: string=?,
15+
) => React.element = "Alert"

0 commit comments

Comments
 (0)