[perf] LTX2 FP4-linear on DGX Spark (GB10/sm_121): ~24% faster denoise at 1080p#1594
[perf] LTX2 FP4-linear on DGX Spark (GB10/sm_121): ~24% faster denoise at 1080p#1594Mister-Raggs wants to merge 3 commits into
Conversation
Env-toggled A/B (FP4_LINEAR, LTX2_HEIGHT/WIDTH/FRAMES/STEPS/NUM_RUNS, LTX2_COMPILE, LTX2_VAE_TILING) that measures per-stage denoise/decode timings for FP4 linear vs bf16 on LTX2.3-distilled. Used to produce the 1080p headline number for the Spark FP4-linear writeup. FP4 linear is already enabled in main via hao-ai-lab#1488.
Match FastVideo's perf-benchmark convention (.buildkite/performance-benchmarks/tests/wan-t2v-1.3b.json) so the harness produces publishable medians out of the box instead of the ad-hoc 4/2.
|
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
Mister-Raggs
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new benchmark script ltx2_fp4_ab.py to compare FP4 quantization against a BF16 baseline for LTX2 video generation. The script handles validation data loading, video generation, and detailed execution stage timing breakdowns. The review feedback focuses on improving the script's robustness, specifically by: 1) falling back to dictionary key lookups when logging_info is a dictionary rather than an object; 2) ensuring torch.cuda.is_available() is checked before synchronizing CUDA; and 3) guarding against potential ZeroDivisionError division-by-zero crashes when calculating average latencies if the number of runs is fewer than the warmup runs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| stages = getattr(logging_info, "stages", None) | ||
| if not stages: |
There was a problem hiding this comment.
If logging_info is a dictionary (which is common when results are serialized or returned as standard dictionaries), getattr(logging_info, "stages", None) will return None. To make the stage breakdown robust to both object and dictionary representations of logging_info, we should fallback to dictionary key lookup if getattr returns None.
| stages = getattr(logging_info, "stages", None) | |
| if not stages: | |
| stages = getattr(logging_info, "stages", None) | |
| if stages is None and isinstance(logging_info, dict): | |
| stages = logging_info.get("stages") | |
| if not stages: |
| stages = getattr(logging_info, "stages", None) | ||
| if not stages: |
There was a problem hiding this comment.
If logging_info is a dictionary, getattr(logging_info, "stages", None) will return None. To make the stage breakdown robust to both object and dictionary representations of logging_info, we should fallback to dictionary key lookup if getattr returns None.
| stages = getattr(logging_info, "stages", None) | |
| if not stages: | |
| stages = getattr(logging_info, "stages", None) | |
| if stages is None and isinstance(logging_info, dict): | |
| stages = logging_info.get("stages") | |
| if not stages: |
| stages = getattr(logging_info, "stages", None) | ||
| if not stages: |
There was a problem hiding this comment.
If logging_info is a dictionary, getattr(logging_info, "stages", None) will return None. To make the stage breakdown robust to both object and dictionary representations of logging_info, we should fallback to dictionary key lookup if getattr returns None.
| stages = getattr(logging_info, "stages", None) | |
| if not stages: | |
| stages = getattr(logging_info, "stages", None) | |
| if stages is None and isinstance(logging_info, dict): | |
| stages = logging_info.get("stages") | |
| if not stages: |
| if os.environ.get("FASTVIDEO_STAGE_LOGGING") == "0": | ||
| torch.cuda.synchronize() |
There was a problem hiding this comment.
For consistency with line 258 and to prevent potential errors on environments where CUDA is not available or initialized, we should check torch.cuda.is_available() before calling torch.cuda.synchronize().
| if os.environ.get("FASTVIDEO_STAGE_LOGGING") == "0": | |
| torch.cuda.synchronize() | |
| if os.environ.get("FASTVIDEO_STAGE_LOGGING") == "0" and torch.cuda.is_available(): | |
| torch.cuda.synchronize() |
| measured_times = run_times[measured_start_idx:] | ||
| avg_time = sum(measured_times) / len(measured_times) | ||
| print( | ||
| f"Average video generation time over {len(measured_times)} runs " | ||
| f"(runs {measured_start_idx + 1}-{len(run_times)}, skipping first {warmup_runs} warmup runs): " | ||
| f"{avg_time:.2f}s" | ||
| ) | ||
|
|
||
| measured_e2e_times = e2e_times[measured_start_idx:] | ||
| avg_e2e_time = sum(measured_e2e_times) / len(measured_e2e_times) | ||
| print( | ||
| f"Average end-to-end latency over {len(measured_e2e_times)} runs " | ||
| f"(runs {measured_start_idx + 1}-{len(e2e_times)}, skipping first {warmup_runs} warmup runs): " | ||
| f"{avg_e2e_time:.2f}s" | ||
| ) |
There was a problem hiding this comment.
If LTX2_NUM_RUNS is configured to be less than or equal to LTX2_WARMUP_RUNS (for example, during a quick test run with LTX2_NUM_RUNS=1 and default LTX2_WARMUP_RUNS=2), measured_times and measured_e2e_times will be empty. This will cause a ZeroDivisionError when calculating avg_time and avg_e2e_time. We should guard these divisions to prevent crashes during quick test runs.
| measured_times = run_times[measured_start_idx:] | |
| avg_time = sum(measured_times) / len(measured_times) | |
| print( | |
| f"Average video generation time over {len(measured_times)} runs " | |
| f"(runs {measured_start_idx + 1}-{len(run_times)}, skipping first {warmup_runs} warmup runs): " | |
| f"{avg_time:.2f}s" | |
| ) | |
| measured_e2e_times = e2e_times[measured_start_idx:] | |
| avg_e2e_time = sum(measured_e2e_times) / len(measured_e2e_times) | |
| print( | |
| f"Average end-to-end latency over {len(measured_e2e_times)} runs " | |
| f"(runs {measured_start_idx + 1}-{len(e2e_times)}, skipping first {warmup_runs} warmup runs): " | |
| f"{avg_e2e_time:.2f}s" | |
| ) | |
| measured_times = run_times[measured_start_idx:] | |
| avg_time = sum(measured_times) / len(measured_times) if measured_times else 0.0 | |
| print( | |
| f"Average video generation time over {len(measured_times)} runs " | |
| f"(runs {measured_start_idx + 1}-{len(run_times)}, skipping first {warmup_runs} warmup runs): " | |
| f"{avg_time:.2f}s" | |
| ) | |
| measured_e2e_times = e2e_times[measured_start_idx:] | |
| avg_e2e_time = sum(measured_e2e_times) / len(measured_e2e_times) if measured_e2e_times else 0.0 | |
| print( | |
| f"Average end-to-end latency over {len(measured_e2e_times)} runs " | |
| f"(runs {measured_start_idx + 1}-{len(e2e_times)}, skipping first {warmup_runs} warmup runs): " | |
| f"{avg_e2e_time:.2f}s" | |
| ) |
There was a problem hiding this comment.
Pull request overview
Adds a new benchmark harness to measure FP4-linear (NVFP4Config) vs bf16 performance for the LTX2.3-distilled inference path at 1080p on DGX Spark (sm_121), with per-stage timing and env-var toggles. This fits into the repo as an examples/-scoped A/B measurement script and does not change production inference code paths.
Changes:
- Add
examples/inference/basic/ltx2_fp4_ab.pyto run a reproducible bf16 vs FP4-linear A/B with stage timing aggregation. - Provide env-var controls for resolution/steps/runs, compile toggle, attention backend override, and VAE tiling safety.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Default 2 warmup + 5 measured (mean), matching FastVideo's perf-benchmark | ||
| # convention (.buildkite/performance-benchmarks/tests/wan-t2v-1.3b.json). | ||
| num_runs = int(os.getenv("LTX2_NUM_RUNS", "7")) | ||
| warmup_runs = int(os.getenv("LTX2_WARMUP_RUNS", "2")) | ||
| avg_window = num_runs - warmup_runs | ||
| measured_start_idx = max(warmup_runs, num_runs - avg_window) | ||
|
|
…afe stages - Raise a clear error when LTX2_WARMUP_RUNS >= LTX2_NUM_RUNS instead of a ZeroDivisionError on an empty measured window; drop the redundant avg_window math (measured_start_idx is always warmup_runs). - Guard the second torch.cuda.synchronize() with torch.cuda.is_available(), matching the warmup-side call. - Read stage timings via a small _stages_of() helper that falls back to dict lookup, so it works whether logging_info is an object or a plain dict.
Purpose
NVFP4 linear quantization already works end-to-end for LTX2.3-distilled after the
fp32→bf16 boundary cast in #1488 — but its payoff on the DGX Spark (GB10, sm_121)
wasn't measured at a headline resolution. This adds the A/B harness that produced
that number and documents the result.
No production code changes — NVFP4 linear runs via flashinfer's
mm_fp4(CUTLASS), which is correct on sm_121, and is enabled purely by attaching
NVFP4Configto the DiT. This PR is the benchmark + evidence.Result — LTX2.3-distilled, 1088×1920×121, 5+2 steps, GB10, SDPA, eager (no compile)
2 warmup + 5 measured runs, mean (matching
.buildkite/performance-benchmarks/tests/wan-t2v-1.3b.json). Same seed/prompt/config across both arms; only the quant config differs.
The win concentrates in the DiT linears (denoise); the VAE side is unchanged, as
expected, so the −39 s e2e ≈ the −40.5 s denoise saving. The relative denoise
speedup is a bit lower at 1080p than at 768×1280 (−24 % vs −30 %) because a larger
share of denoise is attention (SDPA, not quantized) at higher resolution — the
absolute saving is what grows with resolution.
BF16:
bf16_muted.mp4
FP4:
fp4_muted.mp4
Quality
FP4 is lossy; output is visually and audibly equivalent to bf16 at 1080p
(comparison clip + matching-frame stills below). Note SSIM-vs-bf16 is not a
valid quality metric for a 5-step distilled sampler — it reads trajectory
divergence as "loss"; the eye/ear is the correct gate here.
Changes
examples/inference/basic/ltx2_fp4_ab.py— env-toggled FP4-linear-vs-bf16 A/Bharness (per-stage timing,
FP4_LINEAR,LTX2_HEIGHT/WIDTH/FRAMES/STEPS,LTX2_NUM_RUNS/WARMUP,LTX2_COMPILE,LTX2_VAE_TILING). Defaults to the2-warmup/5-measured convention and keeps VAE tiling on (an untiled 1080p decode
OOMs the GB10's unified memory).
Notes / caveats
numbers will differ.
quantizes the linear layers.
torch.compileoff for the A/B to isolate the FP4-linear effect;LTX2_COMPILE=1is available for a tuned run.