Skip to content

Commit 70d968e

Browse files
Bordaclaude[bot]
Borda
and
claude[bot]
authored
feat(cookbooks): add inference latency benchmark notebook (roboflow#1152)
- Added an inference latency benchmark cookbook for FP32, FP16+JIT, and ONNX across detection, segmentation, and keypoint models - Added cookbook guidance for GPU/ONNX setup, CUDA-12 Colab installation, benchmark methodology, and FPS result interpretation - Added a benchmark cookbook card with inference, ONNX, and GPU labels - Added shared ONNX Runtime benchmarking support with explicit provider selection and provider-mismatch failures - Fixed ONNX inference to warn or fail clearly instead of silently falling back to CPU - Fixed PyTorch export loading compatibility for newer PyTorch versions and checkpoints with non-strict key matches - Improved prediction inference performance with inference mode and batched preprocessing while preserving mixed-size image support --------- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
1 parent dc79e78 commit 70d968e

5 files changed

Lines changed: 521 additions & 36 deletions

File tree

‎docs/cookbooks/cards.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
11
cards:
2+
- href: inference-latency-benchmark/
3+
name: "Inference Latency Benchmark"
4+
labels: [INFERENCE, ONNX, GPU]
5+
version: v1.8.1
6+
author: Borda
7+
description: "Benchmark detection, segmentation, and keypoint models across FP32, FP16+JIT, and ONNX Runtime — measures latency and FPS on GPU using CUDA events."
28
- href: fine-tune_keypoints/
39
name: "Fine-Tune RF-DETR Keypoint Detection"
410
labels: [TRAINING, PYTORCH LIGHTNING, INFERENCE]

‎docs/cookbooks/inference-latency-benchmark.ipynb‎

Lines changed: 375 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,375 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"id": "a5720dea",
7+
"metadata": {},
8+
"outputs": [],
9+
"source": [
10+
"# ------------------------------------------------------------------------\n",
11+
"# RF-DETR\n",
12+
"# Copyright (c) 2025 Roboflow. All Rights Reserved.\n",
13+
"# Licensed under the Apache License, Version 2.0 [see LICENSE for details]\n",
14+
"# ------------------------------------------------------------------------"
15+
]
16+
},
17+
{
18+
"cell_type": "markdown",
19+
"id": "66c62889",
20+
"metadata": {},
21+
"source": [
22+
"# RF-DETR Inference Latency Benchmark\n",
23+
"\n",
24+
"Measures inference latency for three RF-DETR families across three configs:\n",
25+
"\n",
26+
"| Config | Description |\n",
27+
"|--------|-------------|\n",
28+
"| **FP32** | `predict()` — unoptimized baseline |\n",
29+
"| **FP16+JIT** | `optimize_for_inference(dtype=torch.float16)` |\n",
30+
"| **ONNX** | exported `.onnx` via `onnxruntime-gpu` |"
31+
]
32+
},
33+
{
34+
"cell_type": "markdown",
35+
"id": "d1568b82",
36+
"metadata": {},
37+
"source": [
38+
"## 1. Install\n",
39+
"\n",
40+
"We need `onnxruntime-gpu` built against CUDA 12 — the default PyPI wheel targets CUDA 11.8 and silently\n",
41+
"falls back to CPU on modern GPUs. The Microsoft CUDA-12 package index ships the correct build.\n",
42+
"\n",
43+
"> **Colab**: after running this cell, go to **Runtime → Restart session**, then run from the next cell."
44+
]
45+
},
46+
{
47+
"cell_type": "code",
48+
"execution_count": null,
49+
"id": "6a116013",
50+
"metadata": {
51+
"title": "[bash]"
52+
},
53+
"outputs": [],
54+
"source": [
55+
"!pip uninstall -y onnxruntime onnxruntime-gpu\n",
56+
"!pip install -q \"rfdetr[onnx]\" pillow pandas\n",
57+
"!pip install -q onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/"
58+
]
59+
},
60+
{
61+
"cell_type": "markdown",
62+
"id": "1ca0838b",
63+
"metadata": {},
64+
"source": [
65+
"## 2. Config\n",
66+
"\n",
67+
"`WARMUP_RUNS` discards the first N inferences — GPU kernels are JIT-compiled on first use, so early timings\n",
68+
"are outliers. `MEASURE_RUNS` then collects the steady-state distribution. 20 + 100 is a reasonable balance\n",
69+
"between statistical stability and total wall-clock time per model."
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": null,
75+
"id": "0e47b983",
76+
"metadata": {},
77+
"outputs": [],
78+
"source": [
79+
"from collections.abc import Callable\n",
80+
"from pathlib import Path\n",
81+
"from typing import Any, NamedTuple\n",
82+
"\n",
83+
"import numpy as np\n",
84+
"import torch\n",
85+
"from PIL import Image\n",
86+
"\n",
87+
"WARMUP_RUNS = 20\n",
88+
"MEASURE_RUNS = 100\n",
89+
"EXPORT_DIR = Path(\"benchmark_output\")\n",
90+
"EXPORT_DIR.mkdir(exist_ok=True)\n",
91+
"\n",
92+
"if not torch.cuda.is_available():\n",
93+
" raise RuntimeError(\"This benchmark requires a CUDA GPU.\")\n",
94+
"print(f\"GPU: {torch.cuda.get_device_name(0)}\")\n",
95+
"\n",
96+
"import onnxruntime as ort\n",
97+
"\n",
98+
"_ort_providers = ort.get_available_providers()\n",
99+
"print(f\"ORT {ort.__version__}, providers: {_ort_providers}\")\n",
100+
"if \"CUDAExecutionProvider\" not in _ort_providers:\n",
101+
" raise RuntimeError(\n",
102+
" f\"onnxruntime-gpu with CUDA support required. Available providers: {_ort_providers}. \"\n",
103+
" \"Fix: reinstall from the CUDA-12 index (see install cell) then restart runtime.\"\n",
104+
" )"
105+
]
106+
},
107+
{
108+
"cell_type": "markdown",
109+
"id": "2ccdc87e",
110+
"metadata": {},
111+
"source": [
112+
"## 3. Sample images\n",
113+
"\n",
114+
"Latency depends on resolution, not pixel content, so synthetic noise images are equivalent to real photos\n",
115+
"for benchmarking purposes. Using a fixed seed makes results reproducible across runs."
116+
]
117+
},
118+
{
119+
"cell_type": "code",
120+
"execution_count": null,
121+
"id": "06a16bb0",
122+
"metadata": {},
123+
"outputs": [],
124+
"source": [
125+
"rng = np.random.default_rng(42)\n",
126+
"images: list[Image.Image] = [Image.fromarray(rng.integers(0, 256, (640, 640, 3), dtype=np.uint8)) for _ in range(10)]\n",
127+
"print(f\"Generated {len(images)} synthetic 640×640 RGB images\")"
128+
]
129+
},
130+
{
131+
"cell_type": "markdown",
132+
"id": "db0d3d7c",
133+
"metadata": {
134+
"lines_to_next_cell": 2
135+
},
136+
"source": [
137+
"## 4. Latency helpers\n",
138+
"\n",
139+
"GPU kernels execute asynchronously — `time.perf_counter()` returns before the GPU finishes, giving\n",
140+
"misleadingly low numbers. CUDA events are inserted directly into the GPU command stream and timestamped\n",
141+
"on the device, so `elapsed_time()` measures actual kernel execution. `torch.cuda.synchronize()` after\n",
142+
"each run flushes the stream and ensures the event fires before we read it."
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": null,
148+
"id": "7f928319",
149+
"metadata": {},
150+
"outputs": [],
151+
"source": [
152+
"class BenchmarkResult(NamedTuple):\n",
153+
" \"\"\"Single benchmark measurement.\"\"\"\n",
154+
"\n",
155+
" label: str\n",
156+
" mean_ms: float\n",
157+
" std_ms: float\n",
158+
"\n",
159+
" @property\n",
160+
" def fps(self) -> float:\n",
161+
" \"\"\"Frames per second.\"\"\"\n",
162+
" return 1000.0 / self.mean_ms\n",
163+
"\n",
164+
"\n",
165+
"def measure_latency_gpu(\n",
166+
" fn: Callable[[], object],\n",
167+
" warmup: int = WARMUP_RUNS,\n",
168+
" runs: int = MEASURE_RUNS,\n",
169+
") -> tuple[float, float]:\n",
170+
" \"\"\"Return (mean_ms, std_ms) using CUDA events.\"\"\"\n",
171+
" for _ in range(warmup):\n",
172+
" fn()\n",
173+
" torch.cuda.synchronize()\n",
174+
" start = torch.cuda.Event(enable_timing=True)\n",
175+
" end = torch.cuda.Event(enable_timing=True)\n",
176+
" timings: list[float] = []\n",
177+
" for _ in range(runs):\n",
178+
" start.record()\n",
179+
" fn()\n",
180+
" end.record()\n",
181+
" torch.cuda.synchronize()\n",
182+
" timings.append(start.elapsed_time(end))\n",
183+
" arr = np.array(timings)\n",
184+
" return float(arr.mean()), float(arr.std())\n",
185+
"\n",
186+
"\n",
187+
"_measure = measure_latency_gpu"
188+
]
189+
},
190+
{
191+
"cell_type": "markdown",
192+
"id": "8f7ee4f5",
193+
"metadata": {
194+
"lines_to_next_cell": 2
195+
},
196+
"source": [
197+
"## 5. Per-config benchmark functions\n",
198+
"\n",
199+
"Three inference paths are compared:\n",
200+
"\n",
201+
"- **FP32** — `predict()` as shipped. Full 32-bit arithmetic on GPU. Baseline.\n",
202+
"- **FP16+JIT** — `optimize_for_inference(dtype=torch.float16)` fuses layers with `torch.jit.script` and\n",
203+
" halves the arithmetic precision. Typically 1.5–2× faster than FP32 with negligible accuracy loss on\n",
204+
" modern tensor cores.\n",
205+
"- **ONNX** — the model is exported to the Open Neural Network Exchange format and run through ONNX Runtime,\n",
206+
" bypassing PyTorch entirely. ORT applies its own graph optimisations and can use the TensorRT execution\n",
207+
" provider for additional speedup. Benchmarked separately on CPU and GPU to show the provider impact."
208+
]
209+
},
210+
{
211+
"cell_type": "code",
212+
"execution_count": null,
213+
"id": "533f037b",
214+
"metadata": {
215+
"lines_to_next_cell": 2
216+
},
217+
"outputs": [],
218+
"source": [
219+
"from rfdetr.export._onnx.inference import _onnx_runtime\n",
220+
"\n",
221+
"\n",
222+
"def _predict_fp32(model: Any, image: Image.Image) -> BenchmarkResult:\n",
223+
" \"\"\"Baseline FP32 predict() latency.\"\"\"\n",
224+
" mean, std = _measure(lambda: model.predict(image))\n",
225+
" return BenchmarkResult(\"predict() FP32\", mean, std)\n",
226+
"\n",
227+
"\n",
228+
"def _predict_fp16(model: Any, image: Image.Image) -> BenchmarkResult:\n",
229+
" \"\"\"FP16+JIT latency — applies and removes optimize_for_inference.\"\"\"\n",
230+
" model.optimize_for_inference(dtype=torch.float16)\n",
231+
" mean, std = _measure(lambda: model.predict(image))\n",
232+
" model.remove_optimized_model()\n",
233+
" return BenchmarkResult(\"predict() FP16+JIT\", mean, std)\n",
234+
"\n",
235+
"\n",
236+
"def _export_onnx(model: Any, export_dir: Path) -> Path:\n",
237+
" \"\"\"Export model to ONNX and return the path.\"\"\"\n",
238+
" return Path(model.export(output_dir=str(export_dir)))"
239+
]
240+
},
241+
{
242+
"cell_type": "markdown",
243+
"id": "2f6bc92a",
244+
"metadata": {
245+
"lines_to_next_cell": 2
246+
},
247+
"source": [
248+
"## 6. Model benchmark runner\n",
249+
"\n",
250+
"Each model is loaded fresh, exported once, and then each inference config is timed independently.\n",
251+
"The ONNX path reuses the same exported file for both CPU and CUDA providers so export cost is not\n",
252+
"counted in latency. CUDA ONNX is skipped gracefully when no GPU is available; missing `onnxruntime-gpu`\n",
253+
"raises immediately so misconfigured environments are caught early."
254+
]
255+
},
256+
{
257+
"cell_type": "code",
258+
"execution_count": null,
259+
"id": "8afd179a",
260+
"metadata": {},
261+
"outputs": [],
262+
"source": [
263+
"def run_model_benchmark(\n",
264+
" model_cls: type,\n",
265+
" model_name: str,\n",
266+
" images: list[Image.Image],\n",
267+
") -> list[BenchmarkResult]:\n",
268+
" \"\"\"Run FP32 / FP16 / ONNX benchmarks for one model and print results.\"\"\"\n",
269+
" print(f\"\\n{'=' * 62}\")\n",
270+
" print(f\" {model_name}\")\n",
271+
" print(\"=\" * 62)\n",
272+
"\n",
273+
" model: Any = model_cls()\n",
274+
" export_dir = EXPORT_DIR / model_name.split()[0]\n",
275+
" export_dir.mkdir(exist_ok=True)\n",
276+
" image = images[0]\n",
277+
"\n",
278+
" fp32 = _predict_fp32(model, image)\n",
279+
" fp16 = _predict_fp16(model, image)\n",
280+
" results: list[BenchmarkResult] = [fp32, fp16]\n",
281+
"\n",
282+
" onnx_path = _export_onnx(model, export_dir)\n",
283+
" for providers in ([\"CPUExecutionProvider\"], [\"CUDAExecutionProvider\", \"CPUExecutionProvider\"]):\n",
284+
" if providers[0] == \"CUDAExecutionProvider\" and not torch.cuda.is_available():\n",
285+
" print(\" ⚠ ONNX (CUDA) skipped — no CUDA GPU\")\n",
286+
" continue\n",
287+
" mean_ms, std_ms, label = _onnx_runtime(onnx_path, image, providers, WARMUP_RUNS, MEASURE_RUNS)\n",
288+
" results.append(BenchmarkResult(f\"ONNX ({label})\", mean_ms, std_ms))\n",
289+
"\n",
290+
" for r in results:\n",
291+
" print(f\" {r.label:<30} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)\")\n",
292+
"\n",
293+
" onnx_results = [r for r in results if r.label.startswith(\"ONNX\")]\n",
294+
" speedups = [f\"FP16 {fp32.mean_ms / fp16.mean_ms:.1f}×\"]\n",
295+
" if onnx_results:\n",
296+
" speedups.append(f\"ONNX {fp32.mean_ms / onnx_results[0].mean_ms:.1f}×\")\n",
297+
" print(f\" Speedup vs FP32: {' | '.join(speedups)}\")\n",
298+
"\n",
299+
" return results"
300+
]
301+
},
302+
{
303+
"cell_type": "markdown",
304+
"id": "e7ccd41c",
305+
"metadata": {},
306+
"source": [
307+
"## 7. Benchmark loop — detection · segmentation · keypoint\n",
308+
"\n",
309+
"Three model families are benchmarked to show how task complexity affects latency. Detection (`RFDETRMedium`)\n",
310+
"outputs boxes only; segmentation (`RFDETRSegSmall`) additionally predicts per-object masks, which adds\n",
311+
"decoder cost; keypoint (`RFDETRKeypointPreview`) predicts skeleton joints and is typically the lightest\n",
312+
"of the three at smaller resolutions."
313+
]
314+
},
315+
{
316+
"cell_type": "code",
317+
"execution_count": null,
318+
"id": "2ab7ffbc",
319+
"metadata": {},
320+
"outputs": [],
321+
"source": [
322+
"from rfdetr import RFDETRKeypointPreview, RFDETRMedium, RFDETRSegSmall\n",
323+
"\n",
324+
"MODELS: list[tuple[type, str]] = [\n",
325+
" (RFDETRMedium, \"RFDETRMedium — detection\"),\n",
326+
" (RFDETRSegSmall, \"RFDETRSegSmall — segmentation\"),\n",
327+
" (RFDETRKeypointPreview, \"RFDETRKeypointPreview — keypoint\"),\n",
328+
"]\n",
329+
"\n",
330+
"all_results: dict[str, list[BenchmarkResult]] = {}\n",
331+
"for _model_cls, _model_name in MODELS:\n",
332+
" all_results[_model_name] = run_model_benchmark(_model_cls, _model_name, images)"
333+
]
334+
},
335+
{
336+
"cell_type": "markdown",
337+
"id": "ebbba970",
338+
"metadata": {},
339+
"source": [
340+
"## 8. Summary\n",
341+
"\n",
342+
"The table shows FPS (frames per second) for each model × config combination. Higher is better.\n",
343+
"Compare columns to see which model fits your latency budget; compare rows to choose the right\n",
344+
"inference backend for your deployment target (Python server, edge CPU, or ONNX Runtime service)."
345+
]
346+
},
347+
{
348+
"cell_type": "code",
349+
"execution_count": null,
350+
"id": "5d8eddd9",
351+
"metadata": {},
352+
"outputs": [],
353+
"source": [
354+
"import pandas as pd\n",
355+
"\n",
356+
"summary = {\n",
357+
" model_name.split()[0]: {r.label: round(r.fps, 1) for r in results} for model_name, results in all_results.items()\n",
358+
"}\n",
359+
"df = pd.DataFrame(summary)\n",
360+
"df.index.name = \"Config \\\\ Model\"\n",
361+
"print(df.to_string())\n",
362+
"print(f\"\\nFPS — {MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, GPU: {torch.cuda.get_device_name(0)}.\")"
363+
]
364+
}
365+
],
366+
"metadata": {
367+
"jupytext": {
368+
"cell_metadata_filter": "title,-all",
369+
"main_language": "python",
370+
"notebook_metadata_filter": "-all"
371+
}
372+
},
373+
"nbformat": 4,
374+
"nbformat_minor": 5
375+
}

0 commit comments

Comments
 (0)