Profile-first CUDA microbenchmark lab for eliminating GPU execution bubbles — stream/event copy–compute overlap, a CUDA Graphs vs torch.compile optimization ladder, and speculative-decoding pipeline overlap, every mechanism verified frame-by-frame in Nsight Systems on RTX 5090 (sm_120).
| Report Bug |
Request Feature |
Laminar is a hands-on microbenchmark lab built on one observation: every framework-side inference optimization is ultimately killing the same thing — GPU execution bubbles, the idle gaps that appear on the GPU timeline when the CPU-side pipeline is drained by a synchronization point.
The toolkit for eliminating them is small and universal:
CUDA Streams (conveyor belts) + Events (timing / handshakes) + async memory copies (move while computing) + CUDA Graphs / torch.compile (kill launch overhead)
Rather than reading about these mechanisms, Laminar builds a minimal reproduction of each one, profiles it in Nsight Systems, applies the fix, and confirms the bubble is gone — turning framework-optimization folklore into muscle memory before diving into real inference engines (SGLang / vLLM).
Environment: RTX 5090 (sm_120) · CUDA 12.8 · torch 2.11 + cu128 · triton 3.6
Each experiment changes exactly one variable, and every claim is backed by an .nsys-rep timeline.
| Experiment | Comparison | Key result | Takeaway |
|---|---|---|---|
| exp0 — tool ladder | operator granularity {small/large} × handling {sync / async / CUDA Graph / torch.compile} | small_sync 2.93 ms → small_graph 2.76 ms → small_compile 2.32 ms; big_* hits a ~1.96 ms floor | Two distinct diseases: launch-bound (cured by fusion / graphs / compile) vs sync-point-bound (cured by async). Graphs only remove launches; compile removes the kernels themselves |
| exp4 — multi-stream overlap | one stream vs D2H copy on a second stream | serial 0.526 ms → overlap 0.312 ms (1.69×) | On a single stream, async only frees the CPU — true overlap requires separate streams (copy engine ⊥ SM, so the parallelism is free) |
| exp5 — speculative-decoding overlap | 6-mode ladder: serial / async_d2h / graph_sync / graph_async / overlap_off / overlap_on | overlap_off 0.260 ms → overlap_on 0.180 ms (1.44×) | Async/overlap is a second-order optimization — invisible until CUDA Graphs remove first-order launch overhead. "Pretend-all-accepted + q·seq = 2" hides verify's D2H under the next draft |
Honest coverage notes (known gaps, kept deliberately visible): no hand-written fused kernel yet (compile's auto-fusion only), no real multi-GPU allgather overlap (single-card lab), and exp5 reproduces the timeline mechanism without token-level rollback logic. See the roadmap.
- build → profile → fix — never optimize what you haven't seen in the timeline; record before/after pairs
- Single-variable discipline — each mode changes exactly one thing
- nsys ≠ ncu — nsys answers when things run (scheduling, overlap, gaps); ncu answers how well a kernel runs (occupancy, Roofline). "Gap-free timeline" and "high SM utilization" are orthogonal goals — this lab targets the former
- Pinned memory +
non_blocking=Trueare a pair — without pinning, async copies silently degrade to blocking and the overlap vanishes - Measurement has its own floor — fixed readback latency can drown real differences; amplify the signal variable before drawing conclusions
Prediction errors are logged, not hidden: docs/design_tracker.md keeps a newest-first record of every prediction, failure, and correction.
Laminar/
├── src/bubble_lab/
│ ├── profiling.py # describe_env() / nvtx_range() / CudaTimer (CUDA-event timing)
│ ├── exp0_tool_ladder.py # fusion / async / CUDA Graph / torch.compile ladder
│ ├── exp4_stream_overlap.py # copy ∥ compute multi-stream overlap
│ └── exp5_spec_overlap.py # speculative-decoding overlap (mainline)
├── scripts/profile.sh # nsys wrapper: bash scripts/profile.sh <module> [args]
├── data/reports/ # captured .nsys-rep timelines
├── docs/design_tracker.md # predictions, failures, corrections — newest first
└── tests/ # profiling-utility tests
- Python 3.11+
uv- CUDA 12.8+ toolchain with Nsight Systems (
nsys) - A PyTorch build matching your GPU architecture (for sm_120, install from the cu128 wheel index before syncing — the resolver must not guess it)
git clone https://github.com/ChaoyuWang04/Laminar_GPU-Bubble-Lab.git
cd Laminar_GPU-Bubble-Lab
uv venv
# install a CUDA-matched torch wheel first, then:
uv pip install -e ".[dev]"Run an experiment directly:
python -m bubble_lab.exp0_tool_ladder --mode all
python -m bubble_lab.exp4_stream_overlap --mode overlap
python -m bubble_lab.exp5_spec_overlap --mode overlap_on --result-mb 8 --steps 10Capture an Nsight Systems timeline for any experiment:
bash scripts/profile.sh bubble_lab.exp5_spec_overlap --mode overlap_on --result-mb 8 --steps 10Reports land in data/reports/ as .nsys-rep files; open them in the Nsight Systems GUI and read the HW rows for gaps.
- Tool ladder: sync / async / CUDA Graph / torch.compile across operator granularities
- Copy ∥ compute multi-stream overlap (1.69×), first HW-row concurrency in nsys
- Speculative-decoding overlap, 6-mode ladder (1.44× end-to-end)
- Shared profiling toolkit (NVTX ranges, CUDA-event timing, env fingerprinting)
- Hand-written Triton fused kernel for exp0 + ncu Roofline analysis of the fusion gap
- Compute ∥ compute dual-stream GEMM (MoE shared-expert prototype) with SM-utilization sampling
- Multi-GPU async allgather overlap (verify whether it back-fires under single-node NVLink)
- Token-level accept/reject logic for exp5, validating equivalence with serial decoding
- Map each mechanism onto its real implementation in SGLang / vLLM source
Issues and pull requests are welcome. The highest-leverage areas:
- additional single-variable bubble reproductions (new sync-point patterns)
- SM-utilization sampling alongside timeline gap analysis
- cross-GPU-generation result comparisons
- Project: https://github.com/ChaoyuWang04/Laminar_GPU-Bubble-Lab
- Author: Chaoyu Wang
Distributed under the MIT License. See LICENSE for more information.