Skip to content

Commit 7b1d38a

Browse files
committed
chore(bench): replace WARP with WAGO runtime
1 parent 8783d9a commit 7b1d38a

9 files changed

Lines changed: 206 additions & 262 deletions

File tree

README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -606,9 +606,9 @@ How `json-as` stacks up against other JSON libraries on a ~5 KiB GitHub-repo pay
606606

607607
### Runtime Comparison
608608

609-
How fast the **same** `json-as` classic bench deserializes the minified payloads across six WebAssembly runtimes — including [WARP](https://github.com/wasm-ecosystem/wasm-compiler), a single-pass compiler built for embedded targets, alongside the optimizing JITs (Wasmtime, WAVM), the pure-Go wazero, and JS engines (V8, Bun).
609+
How fast the **same** `json-as` classic bench deserializes the minified payloads across six WebAssembly runtimes — including [WAGO](https://github.com/wago-org/wago), a pure-Go, no-cgo single-pass JIT, alongside the optimizing JITs (Wasmtime, WAVM), the pure-Go wazero, and JS engines (V8, Bun).
610610

611-
Each runtime runs the real `bench()` lib (warm up, time the loop, report MB/s itself) on a `NAIVE`-mode build with a shared feature set (no SIMD / bulk-memory / non-trapping float-to-int) so the executed code is equivalent. The WASI runtimes read the payload over WASI; V8/Bun via an `env`-ABI host. WARP has no WASI and — by design ("no recursions") — can't re-enter the module from a host import, so it runs through a small custom C++ host that links `performance.now`/`console.log`/`writeFile` with the payload embedded. The timed run is split into small frames with a full GC between them (the bench lib's `BENCH_FRAMES`, applied to every runtime so the measurement is identical); this excludes stop-the-world GC pauses and keeps WARP — an embedded, single-shot-oriented compiler — inside its stable envelope. Even so, WARP's single-pass codegen lands within a few percent of the optimizing JITs.
611+
Each runtime runs the real `bench()` lib (warm up, time the loop, report MB/s itself) on a `NAIVE`-mode build with a shared feature set (no SIMD / bulk-memory / non-trapping float-to-int) so the executed code is equivalent. The WASI runtimes read the payload over WASI; V8/Bun use an `env`-ABI host. WAGO runs through a small Go host built against its public API with the documented `wago_guardpage` mode; the payload is embedded so the comparison does not require a filesystem plugin, while `performance.now`/`console.log`/`writeFile` still keep measurement and result reporting inside the real guest benchmark.
612612

613613
<img src="https://raw.githubusercontent.com/JairusSW/json-as/refs/heads/docs/charts/v1.5.0/runtimes-deserialize.svg" alt="Deserialization throughput across WebAssembly runtimes">
614614

@@ -618,10 +618,10 @@ Each runtime runs the real `bench()` lib (warm up, time the loop, report MB/s it
618618
<img src="https://raw.githubusercontent.com/JairusSW/json-as/refs/heads/docs/charts/v1.5.0/runtimes-serialize.svg" alt="Serialization throughput across WebAssembly runtimes">
619619
</details>
620620

621-
Reproduce locally (the JS engines and standalone runtimes are auto-detected; point `WARP_SRC` at a [wasm-ecosystem/wasm-compiler](https://github.com/wasm-ecosystem/wasm-compiler) checkout with its libs built — see the script header for the cmake flags):
621+
Reproduce locally (the JS engines and standalone runtimes are auto-detected; point `WAGO_SRC` at a [wago-org/wago](https://github.com/wago-org/wago) checkout so the small benchmark host can be built):
622622

623623
```bash
624-
WARP_SRC=/path/to/wasm-compiler npm run bench:runtimes
624+
WAGO_SRC=/path/to/wago npm run bench:runtimes
625625
npm run charts:runtimes
626626
```
627627

assembly/__benches__/lib/bench.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -187,10 +187,10 @@ export function bench(
187187
// Optional framed measurement: split the timed run into BENCH_FRAMES frames
188188
// and run a full __collect() (untimed) between them. Each frame's allocation
189189
// churn stays bounded and the heap is reset between frames, which keeps
190-
// runtimes that destabilize under one long single-shot allocation loop (e.g.
191-
// WARP) inside their safe envelope while still timing the same total ops. The
192-
// between-frame GC pauses are excluded from `elapsed`; the incremental GC that
193-
// runs *within* each frame is still timed, exactly as in the unframed loop.
190+
// memory-constrained or single-shot runtimes inside their safe envelope while
191+
// still timing the same total ops. The between-frame GC pauses are excluded
192+
// from `elapsed`; the incremental GC that runs *within* each frame is still
193+
// timed, exactly as in the unframed loop.
194194
// Defaults to a single frame (identical to the original behavior).
195195
// @ts-expect-error: BENCH_FRAMES may be undefined.
196196
const frames: u64 = isDefined(BENCH_FRAMES) ? u64(BENCH_FRAMES) : 1;

bench/runners/wago_host.go

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
// Command wago_host runs a generated json-as runtime benchmark on WAGO.
2+
//
3+
// The benchmark keeps timing inside WebAssembly. This host only supplies the
4+
// small env ABI used by the AssemblyScript bench library and forwards result
5+
// records to stdout for scripts/run-bench.runtimes.sh.
6+
package main
7+
8+
import (
9+
"encoding/binary"
10+
"fmt"
11+
"os"
12+
"time"
13+
"unicode/utf16"
14+
15+
wago "github.com/wago-org/wago"
16+
)
17+
18+
var epoch = time.Now()
19+
20+
func liftString(m wago.HostModule, ptr uint32) string {
21+
mem := m.Memory()
22+
if ptr == 0 || ptr < 4 || uint64(ptr) > uint64(len(mem)) {
23+
return ""
24+
}
25+
26+
byteLen := binary.LittleEndian.Uint32(mem[ptr-4 : ptr])
27+
end := uint64(ptr) + uint64(byteLen)
28+
if byteLen%2 != 0 || end > uint64(len(mem)) {
29+
return ""
30+
}
31+
32+
units := make([]uint16, byteLen/2)
33+
for i := range units {
34+
off := int(ptr) + i*2
35+
units[i] = binary.LittleEndian.Uint16(mem[off : off+2])
36+
}
37+
return string(utf16.Decode(units))
38+
}
39+
40+
func fail(format string, args ...any) {
41+
fmt.Fprintf(os.Stderr, "wago_host: "+format+"\n", args...)
42+
os.Exit(1)
43+
}
44+
45+
func trace(message string) {
46+
if os.Getenv("WAGO_HOST_TRACE") != "" {
47+
fmt.Fprintln(os.Stderr, "wago_host:", message)
48+
}
49+
}
50+
51+
func main() {
52+
if len(os.Args) != 2 {
53+
fail("usage: wago_host <module.wasm>")
54+
}
55+
56+
wasm, err := os.ReadFile(os.Args[1])
57+
if err != nil {
58+
fail("read module: %v", err)
59+
}
60+
if !wago.GuardPageSupported() {
61+
fail("guard-page bounds are unavailable; build with -tags wago_guardpage")
62+
}
63+
64+
trace("compiling module")
65+
config := wago.NewRuntimeConfig().WithBoundsChecks(wago.BoundsChecksSignalsBased)
66+
compiled, err := wago.Compile(config, wasm)
67+
if err != nil {
68+
fail("compile module: %v", err)
69+
}
70+
defer compiled.Close()
71+
trace("module compiled")
72+
73+
imports := wago.Imports{
74+
"env.performance.now": wago.HostFunc(func(_ wago.HostModule, _ []uint64, results []uint64) {
75+
results[0] = wago.F64(float64(time.Since(epoch).Nanoseconds()) / 1e6)
76+
}),
77+
"env.Date.now": wago.HostFunc(func(_ wago.HostModule, _ []uint64, results []uint64) {
78+
results[0] = wago.F64(float64(time.Now().UnixNano()) / 1e6)
79+
}),
80+
"env.console.log": wago.HostFunc(func(m wago.HostModule, params, _ []uint64) {
81+
fmt.Println(liftString(m, uint32(params[0])))
82+
}),
83+
"env.writeFile": wago.HostFunc(func(m wago.HostModule, params, _ []uint64) {
84+
name := liftString(m, uint32(params[0]))
85+
data := liftString(m, uint32(params[1]))
86+
fmt.Printf("__AS_BENCH_JSON__%s\t%s\n", name, data)
87+
}),
88+
"env.abort": wago.HostFunc(func(m wago.HostModule, params, _ []uint64) {
89+
fmt.Fprintf(
90+
os.Stderr,
91+
"abort: %s in %s:%d:%d\n",
92+
liftString(m, uint32(params[0])),
93+
liftString(m, uint32(params[1])),
94+
uint32(params[2]),
95+
uint32(params[3]),
96+
)
97+
panic(wago.HostExit{Code: 1})
98+
}),
99+
}
100+
101+
trace("instantiating module")
102+
instance, err := wago.Instantiate(compiled, wago.InstantiateOptions{Imports: imports})
103+
if err != nil {
104+
fail("instantiate module: %v", err)
105+
}
106+
defer instance.Close()
107+
trace("module instantiated")
108+
109+
// The module is built with --exportStart so initialization and the full
110+
// benchmark run through WAGO's normal exported-function invocation path.
111+
trace("running benchmark")
112+
if _, err := instance.Invoke("start"); err != nil {
113+
fail("run benchmark: %v", err)
114+
}
115+
trace("benchmark finished")
116+
}

bench/runners/warp_host.cpp

Lines changed: 0 additions & 143 deletions
This file was deleted.

scripts/build-chart-runtimes.ts

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
// Cross-runtime throughput chart for the classic payloads. Compares how fast the
22
// SAME json-as NAIVE-mode bench deserializes the minified payloads under six
3-
// WebAssembly runtimes (WARP / wasmtime / wavm / wazero / v8 / bun). Every bar is
3+
// WebAssembly runtimes (WAGO / wasmtime / wavm / wazero / v8 / bun). Every bar is
44
// the runtime's own bench()-reported MB/s - see scripts/run-bench.runtimes.sh.
55
//
66
// Populate the logs first:
7-
// WARP_SRC=/path/to/wasm-compiler bash scripts/run-bench.runtimes.sh
7+
// WAGO_SRC=/path/to/wago bash scripts/run-bench.runtimes.sh
88
// Then:
99
// bun scripts/build-chart-runtimes.ts
1010
import fs from "node:fs";
@@ -16,11 +16,11 @@ import {
1616
} from "./lib/bench-utils";
1717
import { rgba, BASE } from "./lib/palette";
1818

19-
// One distinct hue per runtime; WARP (the subject) gets the hero blue.
19+
// One distinct hue per runtime; WAGO (the subject) gets the hero blue.
2020
const RUNTIMES: { key: string; label: string; bg: string; border: string }[] = [
2121
{
22-
key: "warp",
23-
label: "WARP",
22+
key: "wago",
23+
label: "WAGO",
2424
bg: rgba("pacificBlue", 0.9),
2525
border: BASE.pacificBlue,
2626
},

scripts/build-charts.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,9 @@ JSON_CHART_RUNTIME="$CHART_RUNTIME" bun ./scripts/build-library-deserialize.ts
5252
bun ./scripts/build-lazy.ts
5353
# Classic-dataset mode comparison (NAIVE/SWAR/SIMD + lazy, no JS baseline)
5454
bun ./scripts/build-chart-classic.ts
55-
# Cross-runtime comparison (WARP/wasmtime/wasmer/wavm/v8/bun). Opt-in: only built
55+
# Cross-runtime comparison (WAGO/wasmtime/wavm/wazero/v8/bun). Opt-in: only built
5656
# when scripts/run-bench.runtimes.sh has produced logs (it needs external
57-
# runtimes + a WARP vb_bench build), so the default chart build never fails on it.
57+
# runtimes), so the default chart build never fails on it.
5858
if compgen -G "./build/logs/runtimes/*/*.deserialize.json" >/dev/null 2>&1; then
5959
bun ./scripts/build-chart-runtimes.ts
6060
fi

scripts/gen-wago-bench.mjs

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
// Generates the WAGO variant of a classic bench. The WAGO host intentionally
2+
// exposes only timing, logging, result writing, and abort imports, so payloads
3+
// are embedded instead of requiring a filesystem plugin. The generated module
4+
// still uses the real bench()/dumpToFile calls and measures the same JSON work.
5+
//
6+
// Usage: node scripts/gen-wago-bench.mjs <name> <out.ts>
7+
import fs from "node:fs";
8+
import path from "node:path";
9+
10+
const [, , name, outPath] = process.argv;
11+
if (!outPath) {
12+
console.error("usage: gen-wago-bench.mjs <name> <out.ts>");
13+
process.exit(1);
14+
}
15+
16+
let src = fs.readFileSync(
17+
path.resolve(`assembly/__benches__/classic/${name}.bench.ts`),
18+
"utf8",
19+
);
20+
21+
// Drop test-only assertions, which are not part of the measured workload.
22+
src = src
23+
.split("\n")
24+
.filter((line) => !/from\s+["'].*__tests__\/lib["']/.test(line))
25+
.filter((line) => !/^\s*expect\(/.test(line))
26+
.join("\n");
27+
28+
// Replace every readFile("<path>") call with the file's escaped contents.
29+
src = src.replace(/readFile\(\s*"([^"]+)"\s*,?\s*\)/g, (_match, filePath) =>
30+
JSON.stringify(fs.readFileSync(path.resolve(filePath), "utf8")),
31+
);
32+
33+
// The other bindings from the bench helper import remain in use.
34+
src = src.replace(/^\s*readFile,\s*$/m, "");
35+
36+
const header = `// AUTO-GENERATED by scripts/gen-wago-bench.mjs - do not edit.
37+
// WAGO build of the "${name}" classic bench: identical schema and bench calls,
38+
// with its payload embedded. Results are emitted through the WAGO host.
39+
`;
40+
41+
fs.writeFileSync(path.resolve(outPath), header + src);
42+
console.log(`> ${outPath}`);

0 commit comments

Comments
 (0)