Skip to content

Commit 2884985

Browse files
yans3metameta-codesync[bot]
yans3meta
authored and
meta-codesync[bot]
committed
Add multi-instance GAPBS (gapbs_multi) with graph generator
Reviewed By: hasan3050 Differential Revision: D113092739 fbshipit-source-id: f22c874fb2c4be30c82f7041e0101c7b4f99a3fe
1 parent 186d153 commit 2884985

7 files changed

Lines changed: 800 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ major production workloads listed as follows:
9999
<td>ffmpeg, svt-av1, libaom, x264</td>
100100
<td>Video Processing </td>
101101
</tr>
102+
<tr>
103+
<td><a href="packages/gapbs/README.md">GAPBS</a> (multi-instance)</td>
104+
<td>C++</td>
105+
<td>GAP Benchmark Suite, OpenMP, systemd cgroups</td>
106+
<td>Graph Analytics, Large / Tiered Memory (CXL) </td>
107+
</tr>
102108
</table>
103109

104110
### AI/ML Benchmarks

‎benchpress/config/benchmarks_mem.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,3 +119,24 @@ gapbs_tc:
119119
- build_time
120120
- trial_time
121121
- average_time
122+
123+
# Multi-instance GAPBS. Launches N core-pinned gapbs instances, each in its own
124+
# systemd-run cgroup scope with per-instance memory.low/high/max limits, on a
125+
# pre-built graph. run-gapbs-multi.sh aggregates each instance's Average Time
126+
# (mean/min/max). Build graphs with ./benchmarks/gapbs/generate_graph.sh.
127+
gapbs_multi:
128+
parser: gapbs
129+
install_script: ./packages/gapbs/install_gapbs.sh
130+
cleanup_script: ./packages/gapbs/cleanup_gapbs.sh
131+
install_markers:
132+
- ./benchmarks/gapbs/bc
133+
- ./benchmarks/gapbs/run-gapbs-multi.sh
134+
path: ./benchmarks/gapbs/run-gapbs-multi.sh
135+
tags:
136+
scope:
137+
- kernel
138+
component:
139+
- cpu
140+
- mem
141+
metrics:
142+
- average_time

‎benchpress/config/jobs_mem.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,3 +84,56 @@
8484
vars:
8585
- "scale=25"
8686
- "trials=3"
87+
88+
# Multi-instance GAPBS on a PRE-BUILT graph. Launches {num_instances} core-pinned
89+
# gapbs instances of {bench}, each in its own systemd-run cgroup scope with
90+
# per-instance memory.low/high/max limits, then aggregates each instance's
91+
# Average Time (mean/min/max).
92+
#
93+
# Requirements / notes:
94+
# * The graph must already exist. Build it once with:
95+
# ./benchmarks/gapbs/generate_graph.sh -g 30 -k 33
96+
# (generates kronecker30_k33.sg; ~350 GB, needs >=500 GB free.)
97+
# * Per-instance memory limits require running benchpress as root; without
98+
# root the runner falls back to taskset pinning WITHOUT memory limits.
99+
# * mem_low/high/max accept human suffixes (e.g. 290G) or 'max' (unlimited).
100+
# * Core layout counts PHYSICAL cores; SMT siblings are auto-added to the
101+
# reserved cpuset. The kernel runs on threads_per_instance cores.
102+
# * bench is one of: bc bfs cc pr sssp tc. graph_file is a base name (no
103+
# extension) resolved per kernel: bc/bfs/cc/pr->.sg, sssp->.wsg, tc->U.sg.
104+
- benchmark: gapbs_multi
105+
name: gapbs_multi
106+
description: >
107+
Multi-instance GAP benchmark. Launches {num_instances} core-pinned gapbs
108+
instances, each in a systemd-run cgroup scope with per-instance
109+
memory.low/high/max limits, on a pre-built graph. Aggregates per-instance
110+
Average Time (mean/min/max).
111+
args:
112+
- '-n {num_instances}'
113+
- '-b {bench}'
114+
- '-f {graph_file}'
115+
- '-t {trials}'
116+
- '--core-start {core_start}'
117+
- '-c {cores_per_instance}'
118+
- '--core-stride {core_stride}'
119+
- '-T {threads_per_instance}'
120+
- '--mem-low {mem_low}'
121+
- '--mem-high {mem_high}'
122+
- '--mem-max {mem_max}'
123+
- '{extra_args}'
124+
vars:
125+
- "num_instances=3"
126+
- "bench=bc"
127+
- "graph_file=kronecker30_k33"
128+
- "trials=2"
129+
- "core_start=0"
130+
- "cores_per_instance=32"
131+
- "core_stride=32"
132+
- "threads_per_instance=16"
133+
- "mem_low=290G"
134+
- "mem_high=300G"
135+
- "mem_max=310G"
136+
# extra_args is passed verbatim, e.g. "--ccx" to bind each instance to an
137+
# auto-detected CCX (sysfs L3 domain), "--mem-nodes 0,1" for DRAM+CXL, or
138+
# "--no-cgroup" to disable scopes/limits (taskset-only).
139+
- "extra_args="

‎packages/gapbs/README.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
<!--
2+
Copyright (c) Meta Platforms, Inc. and affiliates.
3+
4+
This source code is licensed under the MIT license found in the
5+
LICENSE file in the root directory of this source tree.
6+
-->
7+
# GAPBS (multi-instance)
8+
9+
GAPBS is the [GAP Benchmark Suite](https://github.com/sbeamer/gapbs) — a set of
10+
graph-analytics kernels (`bc`, `bfs`, `cc`, `pr`, `sssp`, `tc`). This package
11+
adds `gapbs_multi`, a memory-focused, multi-instance driver: it launches N
12+
core-pinned gapbs instances of one kernel in parallel against a **pre-built**
13+
graph, each instance in its own cgroup with per-instance memory limits, and
14+
aggregates the per-instance `Average Time` (mean / min / max).
15+
16+
It is intended for large-memory / tiered-memory (e.g. CXL) characterization,
17+
where several graph-analytics instances run side by side under controlled cpu
18+
and memory budgets.
19+
20+
> **Note:** `gapbs_multi` lives in the `mem` benchmark suite, so every command
21+
> must pass `-b mem`.
22+
23+
## Install
24+
25+
```
26+
./benchpress_cli.py -b mem install gapbs_multi
27+
```
28+
29+
This builds the gapbs kernels + `converter` and copies `run-gapbs-multi.sh` and
30+
`generate_graph.sh` into `./benchmarks/gapbs/`.
31+
32+
## Generate the graph
33+
34+
`gapbs_multi` never generates graphs at run time — build one once with the
35+
`generate_graph.sh` helper. The default reproduces `kronecker30_k33.sg`
36+
(a Kronecker graph with 2^30 vertices and average degree 33, ~350 GB):
37+
38+
```
39+
./benchmarks/gapbs/generate_graph.sh -g 30 -k 33
40+
```
41+
42+
The script **aborts unless the output filesystem has at least 500 GB free**
43+
(override with `--min-free-gb`). Useful options:
44+
45+
- `-g <scale>` / `-k <degree>`: graph size (2^scale vertices, avg degree)
46+
- `-b <name|file>`: output base name or path (default `kronecker<scale>_k<degree>`)
47+
- `-d <dir>`: output directory (default `./benchmarks/gapbs`)
48+
- `--weighted`: build a weighted graph (`.wsg`, required for `sssp`)
49+
- `--undirected`: symmetrize + `U` suffix (`.U.sg`, required for `tc`)
50+
- `--min-free-gb <n>`: change the disk-space guard (default 500)
51+
- `--force`: regenerate even if the target already exists
52+
53+
See `./benchmarks/gapbs/generate_graph.sh -h` for the full list.
54+
55+
## Run
56+
57+
### Recommended job - `gapbs_multi`
58+
59+
```
60+
./benchpress_cli.py -b mem run gapbs_multi
61+
```
62+
63+
Each instance is launched as a transient `systemd-run --scope`, i.e. its **own
64+
cgroup**, with `AllowedCPUs` (reserved cores) and `MemoryLow` / `MemoryHigh` /
65+
`MemoryMax` limits. The kernel then runs on `threads_per_instance` cores (exact
66+
OpenMP thread count).
67+
68+
> **Root required:** per-instance memory limits need root (to create the scope).
69+
> Without root, the runner warns and falls back to `taskset` pinning **without**
70+
> memory limits.
71+
72+
This job has the following optional parameters (override with `-i '{...}'`):
73+
74+
- `num_instances`: number of parallel instances (default `3`)
75+
- `bench`: kernel to run — `bc` | `bfs` | `cc` | `pr` | `sssp` | `tc` (default `bc`)
76+
- `graph_file`: graph base name (no extension) or full `.sg`/`.wsg` path,
77+
resolved per kernel: `bc/bfs/cc/pr` → `.sg`, `sssp` → `.wsg`, `tc` → `U.sg`
78+
(default `kronecker30_k33`)
79+
- `trials`: timed trials, passed to gapbs `-n` (default `2`)
80+
- `core_start` / `cores_per_instance` / `core_stride`: physical-core layout
81+
(defaults `0` / `32` / `32`). Counts are **physical cores**; SMT siblings are
82+
auto-added to each instance's reserved cpuset.
83+
- `threads_per_instance`: active OpenMP thread count per instance (default `16`)
84+
- `mem_low` / `mem_high` / `mem_max`: per-instance memory limits, accept human
85+
sizes (e.g. `290G`) or `max` = unlimited (defaults `290G` / `300G` / `310G`)
86+
- `extra_args`: passed verbatim to the runner (see below)
87+
88+
Available `extra_args` (also visible via `./benchmarks/gapbs/run-gapbs-multi.sh -h`):
89+
90+
- `--ccx`: bind instance *i* to auto-detected CCX[*i*] (sysfs L3 domain)
91+
instead of the explicit stride layout
92+
- `--mem-nodes 0,1`: set `cpuset.mems` / `AllowedMemoryNodes` (e.g. DRAM+CXL)
93+
- `--no-cgroup`: disable systemd scopes and memory limits (taskset only)
94+
95+
### Examples
96+
97+
Run **4 instances, each in its own cgroup** (needs root; default layout reserves
98+
physical cores `0-31 / 32-63 / 64-95 / 96-127`, so ≥128 physical cores):
99+
100+
```
101+
sudo ./benchpress_cli.py -b mem run gapbs_multi -i '{"num_instances": 4}'
102+
```
103+
104+
Run 4 instances on a smaller host (16 reserved cores / 8 threads each):
105+
106+
```
107+
sudo ./benchpress_cli.py -b mem \
108+
-o 'gapbs_multi: -n 4 -b bc -f kronecker30_k33 -t 2 -c 16 --core-stride 16 -T 8 --mem-low 290G --mem-high 300G --mem-max 310G' \
109+
run gapbs_multi
110+
```
111+
112+
Bind each instance to a detected CCX and place memory on DRAM+CXL (nodes 0,1):
113+
114+
```
115+
sudo ./benchpress_cli.py -b mem run gapbs_multi \
116+
-i '{"num_instances": 4, "extra_args": "--ccx --mem-nodes 0,1"}'
117+
```
118+
119+
Run the `bfs` kernel with 30 trials:
120+
121+
```
122+
sudo ./benchpress_cli.py -b mem run gapbs_multi -i '{"bench": "bfs", "trials": 30}'
123+
```
124+
125+
## Core layout
126+
127+
For the default (stride) layout, instance *i* reserves the physical cores
128+
129+
```
130+
[core_start + i*core_stride, core_start + i*core_stride + cores_per_instance - 1]
131+
```
132+
133+
plus each of those cores' SMT siblings — this becomes the cgroup `AllowedCPUs`.
134+
The kernel actually runs on only `threads_per_instance` physical cores (one
135+
logical CPU per core), enforced with `taskset` + `OMP_NUM_THREADS`, matching the
136+
"reserved cores vs. active threads" split used for graph-analytics runs.
137+
138+
With `--ccx`, the reserved cpuset is instead a full auto-detected CCX (the L3
139+
`shared_cpu_list` from sysfs); instance *i* maps to CCX[*i*], and the run fails
140+
if there are fewer CCXs than instances.
141+
142+
## Reporting and Measurement
143+
144+
`gapbs_multi` prints an aggregate block to stdout and reports it in JSON. The
145+
headline metric is `average_time` — the **mean** of the per-instance
146+
`Average Time` values (the runner also prints `average_time_min` /
147+
`average_time_max` / `per_instance_average_time` for reference):
148+
149+
```
150+
=== gapbs_multi aggregate (bc, 4 instances) ===
151+
per_instance_average_time: 3.10 3.22 3.05 3.18
152+
average_time_mean=3.137500
153+
average_time_min=3.050000
154+
average_time_max=3.220000
155+
spawned_instances=4
156+
successful_instances=4
157+
Average Time: 3.137500
158+
```
159+
160+
Result locations:
161+
162+
- Parsed metrics JSON: `benchmark_metrics_<run_id>/gapbs_multi_metrics_<timestamp>_iter_*.json`
163+
(plus `gapbs_multi_system_specs_<timestamp>.json`), under the benchpress root
164+
or the directory given by `--artifacts-dir`
165+
- Run history: `./results` (or the directory given by `-r`)
166+
- Per-instance raw kernel logs (on the host): `/tmp/gapbs_multi_<bench>/benchlog_inst*.txt`
167+
168+
To collect system / microarchitecture metrics during the run, add hooks with
169+
`-k`, e.g. `-k cpu-mpstat perf`. These are host-wide, time-window collectors and
170+
cover all instances for the full duration of the run.
171+
172+
## Runner help
173+
174+
```
175+
./benchmarks/gapbs/run-gapbs-multi.sh -h
176+
```

0 commit comments

Comments
 (0)