Skip to content

Commit e268c4f

Browse files
endolinbotclaude
endolinbot
and
claude
committed
xst-release-ab: confirm flat/flatMap overflow on stock release xst before/after 73aad47b
Verifies the ymax0 v320 XS value-stack overflow (#9) on the stock upstream xst shell built from Moddable release tags, per mhofman's ask to confirm with release xst instead of the agoric worker. Leaf (hex.js flatMap) class overflows on 7.0.0, clears on 7.1.0/8.3.0 (73aad47b's leaf pop). The nested branch still overflows on the latest release (8.3.0) because 73aad47b pops only the leaf slot; the overflow moves out from N~150K to N~256K rather than disappearing, which kriscendobot/moddable#1's end-of-block pop closes. Includes build-xst.sh, run-ab.sh, and two Moddable-adoptable test cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9964115 commit e268c4f

5 files changed

Lines changed: 247 additions & 0 deletions

File tree

‎skills/agoric-chain-snapshot/repro/xst-release-ab/README.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Stock-`xst` A/B — flat/flatMap value-stack overflow before/after `73aad47b`
2+
3+
This harness answers mhofman's follow-up on kriskowal/garden#9
4+
([comment 4907678857](https://github.com/kriskowal/garden/issues/9#issuecomment-4907678857)):
5+
6+
> Can you verify the stack overflow using `xst` instead of our worker? You should
7+
> be able to use release `xst`, from before and after `73aad47b` to confirm. If
8+
> the pop at the end of the block is indeed the more correct place, Moddable needs
9+
> a test case to reproduce the issue against the latest release.
10+
11+
Everything here runs on the **stock upstream `xst`** (the standalone XS shell) built
12+
straight from tagged **Moddable-OpenSource/moddable releases** — no agoric fork, no
13+
xsnap worker, no patches. It complements `../engine-flatmap-ab/` (which A/B's the
14+
three `mxPop()` placements on the agoric xsnap worker at the on-chain
15+
`stackCount=4096`); this one confirms the same behavior on the canonical engine
16+
release binaries, at `xst`'s own default 256K-slot value stack.
17+
18+
## The release window
19+
20+
[`Moddable-OpenSource/moddable@73aad47b`](https://github.com/Moddable-OpenSource/moddable/commit/73aad47b3eb5f5f13baf401bd28d1609c14f23ab)
21+
("XS: fix array flat/sort stack overflows") landed **2026-01-20**.
22+
23+
| release | date | contains `73aad47b`? | `flatAux` leaf branch |
24+
| --- | --- | --- | --- |
25+
| **7.0.0** | 2026-01-16 | **no** (BEFORE) | `mxDefineIndex; start++` — no pop |
26+
| **7.1.0** | 2026-02-11 | yes (first AFTER) | `mxDefineIndex; mxPop(); start++` |
27+
| **8.3.0** | 2026-07-03 | yes (LATEST) | `mxDefineIndex; mxPop(); start++` |
28+
29+
Confirmed with `gh api .../compare/73aad47b...<tag>`: 7.0.0 is *behind* the commit
30+
(does not contain it), 7.1.0 and 8.3.0 are *ahead* (contain it).
31+
32+
## The two branches of `fx_Array_prototype_flatAux`
33+
34+
`fx_Array_prototype_flatAux` (`xs/sources/xsArray.c`) builds the flattened result on
35+
the heap but, in stock XS, leaves per-iteration slots resident on the **value
36+
stack**, so peak use is O(flattened output), not O(depth). Two leak sites:
37+
38+
- **LEAF branch** — `item` defined into the result via `mxDefineIndex`; `fxDefineAll`
39+
pops only the result-array reference, leaving the value below it. `73aad47b` adds
40+
`mxPop()` here. This is the ymax0 `hex.js` class (`new Map(RI.flatMap(...))`,
41+
leaf-dominated).
42+
- **NESTED branch** — `item = the->stack` (the sub-array), pushed before descending;
43+
after the recursion returns it is **never popped**. `73aad47b` does **not** touch
44+
this branch, so it still leaks one slot per nested element on the latest release.
45+
kriscendobot/moddable#1 moves the pop to the **end of the `if (fxHasIndex)` block**,
46+
covering both branches.
47+
48+
## Build the three release `xst`s
49+
50+
```sh
51+
for tag in 7.0.0 7.1.0 8.3.0; do ./build-xst.sh $tag; done
52+
# each prints .../src/moddable-<tag>/build/bin/lin/release/xst
53+
```
54+
55+
## Run the A/B
56+
57+
```sh
58+
XST() { echo ~/.cache/garden-scratch/xst-ab/src/moddable-$1/build/bin/lin/release/xst; }
59+
# LEAF (ymax0 hex.js class) — fast:
60+
for tag in 7.0.0 7.1.0 8.3.0; do ./run-ab.sh leaf 300000 "$(XST $tag)"; done
61+
# NESTED (the branch 73aad47b leaves resident) — SLOW at 256K stack (see note):
62+
for tag in 7.0.0 7.1.0 8.3.0; do ./run-ab.sh nested 300000 "$(XST $tag)" 600; done
63+
```
64+
65+
## Verified results (2026-07-07, x86-64 linux, GOAL=release)
66+
67+
Stock `xst` runs a script file with a **256K-slot** value stack
68+
(`xs/tools/xst.c` `_creation.stackCount = 256*1024`), so thresholds are ~256K
69+
elements — far above the on-chain xsnap 4096, but the **same leak**. A value-stack
70+
exhaustion surfaces as `Error: JavaScript stack overflow` (exit 1).
71+
72+
### LEAF — `[inner].flat()`, inner = K scalars (the ymax0 `flatMap` class)
73+
74+
| K | 7.0.0 (before) | 7.1.0 (after) | 8.3.0 (latest) |
75+
| --- | --- | --- | --- |
76+
| 300000 | **STACK OVERFLOW** | OK | OK |
77+
78+
Fast (< 1 s): the leaf loop allocates almost nothing, so it fills the stack in one
79+
near-linear pass. **`73aad47b` fixes the ymax0-class (leaf) overflow on release
80+
`xst`, before → overflow, after → clean.** This is the decisive "verify with `xst`,
81+
before and after" result.
82+
83+
### NESTED — `a = [inner,inner,…] (N×); a.flat(1)`, every element an array
84+
85+
| N | 7.0.0 (before) | 7.1.0 (after) | 8.3.0 (latest) |
86+
| --- | --- | --- | --- |
87+
| 130000 | OK (42 s) | — | — |
88+
| 160000 | **STACK OVERFLOW** (76 s) | — | — |
89+
| 300000 | **STACK OVERFLOW** | **STACK OVERFLOW** (229 s) | **STACK OVERFLOW** (218 s) |
90+
91+
Before the fix the nested case leaks **two** slots per element (leaf + sub-array), so
92+
it overflows at N ≈ 150K. After `73aad47b` the leaf slot is popped but the sub-array
93+
slot is **not**, so the leak halves (one slot/element) and the overflow simply moves
94+
out to N ≈ 256K — **it does not go away on the latest release.** That is the case
95+
`73aad47b` leaves open and kriscendobot/moddable#1's end-of-block pop closes.
96+
97+
> **Timing note.** At the 256K stock stack the nested overflow is O(N²) to reach —
98+
> XS's GC rescans the growing value stack on every collection — so each nested run
99+
> is minutes, not seconds. This is a property of the large default stack, not the
100+
> bug: at the on-chain `stackCount=4096` the identical nested overflow trips at
101+
> N ≈ 4200 in well under a second (`../engine-flatmap-ab/README.md`).
102+
103+
## Test cases for Moddable
104+
105+
- [`testcase-flat-leaf-overflow.js`](./testcase-flat-leaf-overflow.js) — reproduces
106+
the overflow `73aad47b` fixed; a regression guard for the leaf pop (overflows
107+
≤ 7.0.0, passes ≥ 7.1.0).
108+
- [`testcase-flat-nested-overflow.js`](./testcase-flat-nested-overflow.js) — **the
109+
one mhofman asked for**: still overflows on the **latest** release `xst` (8.3.0),
110+
because 73aad47b's leaf-only pop leaves the nested branch resident. Passes only
111+
with kriscendobot/moddable#1's end-of-block pop. Runs in minutes at the 256K stock
112+
stack; Moddable's harness can create the machine with a small `stackCount` (e.g.
113+
4096) to trip it in well under a second.
114+
115+
## Conclusion
116+
117+
1. **Verified on release `xst`, before and after `73aad47b`:** 7.0.0 overflows on the
118+
ymax0 (leaf) class, 7.1.0 and 8.3.0 clear it. The cherry-pick is confirmed on the
119+
canonical engine, not just the agoric worker.
120+
2. **The end-of-block pop is the more correct place, confirmed on release `xst`:** the
121+
nested branch still leaks on the latest release (8.3.0) — its overflow only moves
122+
out from N ≈ 150K to N ≈ 256K, it does not disappear. `testcase-flat-nested-overflow.js`
123+
is the reproduction against the latest release Moddable can adopt.
124+
125+
Scope: read-only analysis + on-host builds/runs of the open-source XS engine from
126+
public Moddable release tarballs. No upstream `Moddable-OpenSource/moddable` or
127+
`agoric/agoric-sdk` interaction.

‎skills/agoric-chain-snapshot/repro/xst-release-ab/build-xst.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
# Build the stock upstream `xst` (the standalone XS shell) from a tagged
3+
# Moddable-OpenSource/moddable RELEASE — no agoric fork, no patches. This is the
4+
# engine mhofman asked to confirm the ymax0 flat/flatMap value-stack overflow
5+
# against (kriskowal/garden#9, comment 4907678857): "verify the stack overflow
6+
# using xst instead of our worker … release xst, from before and after 73aad47b".
7+
#
8+
# 73aad47b ("XS: fix array flat/sort stack overflows") landed 2026-01-20, between
9+
# release 7.0.0 (2026-01-16, BEFORE — no fix) and 7.1.0 (2026-02-11, AFTER — has
10+
# the leaf-only pop). 8.3.0 (2026-07-03) is the latest release and also carries
11+
# only the leaf-only pop.
12+
#
13+
# Usage: build-xst.sh <tag> e.g. build-xst.sh 7.0.0
14+
# Output: prints the absolute path to the built xst binary.
15+
set -euo pipefail
16+
tag="${1:?usage: build-xst.sh <tag> (e.g. 7.0.0 | 7.1.0 | 8.3.0)}"
17+
WORK="${XST_AB_WORK:-$HOME/.cache/garden-scratch/xst-ab}"
18+
src="$WORK/src/moddable-$tag"
19+
mkdir -p "$WORK/src"
20+
if [ ! -d "$src" ]; then
21+
curl -sL "https://github.com/Moddable-OpenSource/moddable/archive/refs/tags/$tag.tar.gz" \
22+
-o "$WORK/src/$tag.tar.gz"
23+
tar xzf "$WORK/src/$tag.tar.gz" -C "$WORK/src"
24+
fi
25+
# xst must live on an exec filesystem; the release build tree is fine.
26+
( cd "$src/xs/makefiles/lin" && MODDABLE="$src" make -f xst.mk GOAL=release >/dev/null 2>&1 )
27+
bin="$src/build/bin/lin/release/xst"
28+
[ -x "$bin" ] || { echo "build failed: $bin missing" >&2; exit 1; }
29+
echo "$bin"

‎skills/agoric-chain-snapshot/repro/xst-release-ab/run-ab.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/bin/bash
2+
# Run one flat/flatMap value-stack probe through a chosen stock `xst` and
3+
# classify the result OK / OVERFLOW / TIMEOUT with wall-clock timing.
4+
#
5+
# Two shapes, isolating the two leak branches of fx_Array_prototype_flatAux
6+
# (xs/sources/xsArray.c):
7+
# leaf — `[inner].flat()` where inner is K scalars: one nested descent, then
8+
# K LEAF defines. 73aad47b pops the leaf slot, so this is the branch
9+
# the fix covers. Mirrors the ymax0 hex.js `new Map(RI.flatMap(...))`
10+
# class (leaf-dominated).
11+
# nested — `Array.from({length:N},()=>[{}]).flat(1)`: every element hits the
12+
# NESTED branch, whose `item` slot 73aad47b does NOT pop. This is the
13+
# branch only kriscendobot/moddable#1's end-of-block pop covers.
14+
#
15+
# Stock `xst` runs a script file with a 256K-slot value stack (xs/tools/xst.c
16+
# _creation.stackCount = 256*1024), so thresholds are ~256K elements — far above
17+
# the on-chain xsnap 4096, but the SAME leak. The engine-flatmap-ab harness one
18+
# dir up shows the same A/B at the on-chain stackCount=4096.
19+
#
20+
# Usage: run-ab.sh <leaf|nested> <N> <path-to-xst> [timeout_s]
21+
set -uo pipefail
22+
shape="${1:?shape}"; N="${2:?N}"; xst="${3:?xst path}"; to="${4:-240}"
23+
T="$(mktemp -d)"
24+
if [ "$shape" = leaf ]; then
25+
cat > "$T/run.js" <<EOF
26+
var K=$N; var inner=[]; for(var i=0;i<K;i++)inner.push(i);
27+
var r=[inner].flat(); print("OK leaf K="+K+" len="+r.length);
28+
EOF
29+
else
30+
cat > "$T/run.js" <<EOF
31+
var N=$N; var a=Array.from({length:N},function(){return [{}];});
32+
var r=a.flat(1); print("OK nested N="+N+" len="+r.length);
33+
EOF
34+
fi
35+
s=$SECONDS
36+
out="$(timeout "$to" "$xst" "$T/run.js" 2>&1)"; rc=$?
37+
dt=$((SECONDS-s))
38+
cls=UNKNOWN
39+
[ "$rc" -eq 0 ] && cls=OK
40+
echo "$out" | grep -qi 'stack overflow' && cls=OVERFLOW
41+
[ "$rc" -eq 124 ] && cls=TIMEOUT
42+
printf '%-6s N=%-7s %-8s exit=%-3s %ss :: %s\n' "$shape" "$N" "$cls" "$rc" "$dt" "$(echo "$out" | head -1)"

‎skills/agoric-chain-snapshot/repro/xst-release-ab/testcase-flat-leaf-overflow.js‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
// Moddable test case — Array.prototype.flat/flatMap value-stack overflow (LEAF).
2+
//
3+
// This is the ymax0 v320 class: `new Map(RI.flatMap(...))` in @agoric/internal
4+
// hex.js. fx_Array_prototype_flatAux (xs/sources/xsArray.c) builds the flattened
5+
// result on the heap but, in stock XS, leaves each defined LEAF element resident
6+
// on the value stack — peak value-stack use is O(flattened output), not O(depth).
7+
//
8+
// EXPECTED:
9+
// * BEFORE 73aad47b (release 7.0.0 and earlier): "Error: JavaScript stack
10+
// overflow" — the K leaf defines pile up on the value stack.
11+
// * AFTER 73aad47b (release 7.1.0 .. 8.3.0): completes; prints "PASS".
12+
//
13+
// The single outer element makes exactly one nested descent, then the inner
14+
// array's K elements all hit the LEAF branch — the branch 73aad47b's mxPop()
15+
// covers. K is sized above xst's 256K-slot value stack (xs/tools/xst.c
16+
// _creation.stackCount = 256*1024). On the on-chain xsnap stackCount=4096 the
17+
// same overflow trips at K ~ 4000.
18+
var K = 300000;
19+
var inner = [];
20+
for (var i = 0; i < K; i++) inner.push(i);
21+
var r = [inner].flat();
22+
if (r.length !== K) throw new Error("wrong length " + r.length);
23+
print("PASS flat leaf K=" + K);

‎skills/agoric-chain-snapshot/repro/xst-release-ab/testcase-flat-nested-overflow.js‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
// Moddable test case — Array.prototype.flat value-stack overflow (NESTED branch).
2+
//
3+
// This is the case that STILL OVERFLOWS on the LATEST release (8.3.0), because
4+
// 73aad47b only pops in the LEAF branch of fx_Array_prototype_flatAux; the NESTED
5+
// branch (xs/sources/xsArray.c) fetches `item = the->stack` and, after recursing
6+
// into the sub-array, never pops it — so each nested element leaks one value-stack
7+
// slot even with the fix. kriscendobot/moddable#1 moves the mxPop() to the END of
8+
// the `if (fxHasIndex)` block, covering BOTH branches, and clears this case.
9+
//
10+
// EXPECTED:
11+
// * stock XS through 8.3.0 (leaf-only pop): "Error: JavaScript stack overflow".
12+
// * with kriscendobot/moddable#1's end-of-block pop: completes, prints "PASS".
13+
//
14+
// Every element of `a` is an array, so every element hits the NESTED branch.
15+
// N is sized above xst's 256K-slot value stack. NOTE: at the 256K stock stack this
16+
// runs for a few minutes — reaching the overflow is O(N^2) because XS's GC rescans
17+
// the growing value stack. Moddable's own harness can create the machine with a
18+
// small stackCount (e.g. the on-chain xsnap 4096), where the identical overflow
19+
// trips at N ~ 4200 in well under a second (see engine-flatmap-ab/README.md).
20+
var N = 300000;
21+
var inner = [0];
22+
var a = new Array(N);
23+
for (var i = 0; i < N; i++) a[i] = inner;
24+
var r = a.flat(1);
25+
if (r.length !== N) throw new Error("wrong length " + r.length);
26+
print("PASS flat nested N=" + N);

0 commit comments

Comments
 (0)