[Perf]: MOSS-Transcribe-Diarize model Encoder torch.compile

[nagisa-kunhah]

Motivation

#924

Add and evaluate optional torch.compile support for the MOSS Transcribe-Diarize ASR encoder path. The goal is to check whether compiling only the encoder can improve end-to-end latency without introducing other optimization changes.

Modifications

• Add an encoder-only benchmark:
  • benchmarks/eval/benchmark_moss_transcribe_diarize_encoder_compile.py
• Document the benchmark method and measured e2e results.
• Use max-autotune-no-cudagraphs with dynamic=False as the recommended encoder compile configuration.
• Keep the benchmark scope limited to the Whisper encoder path compiled by compile_audio_encoder.
• Exclude VQ adaptor, language model, router/HTTP layer, cache, request preprocessing, concurrency tuning, and other service-side optimizations from the benchmark scope.

Benchmark & Profiling

Encoder-only benchmark

Environment:

• GPU: NVIDIA H800 PCIe, 80GB
• Model: /root/models/MOSS-Transcribe-Diarize
• Script: benchmarks/eval/benchmark_moss_transcribe_diarize_encoder_compile.py
• Config: --frames 3000 --warmup 5 --iters 50
• Compile mode: max-autotune-no-cudagraphs
• dynamic=False

Mode	mean ms	p50 ms	p95 ms	speedup vs eager
eager	6.9335	6.3678	8.3071	1.0000x
torch.compile:max-autotune-no-cudagraphs	5.1551	5.1053	5.1569	1.3450x

The encoder hot path improves from 6.9335 ms to 5.1551 ms mean latency, about 1.345x faster.

E2E benchmark: short audio

Config:

• Audio: results/query_to_cars_30s.wav
• Total audio duration: 332.64s
• Requests: 12
• Concurrency: 1
• Warmup: 1

Mode	success	throughput req/s	latency mean s	latency p50 s	latency p95 s
encoder compile off	12/12	5.3548	0.1864	0.1909	0.2199
encoder compile on	12/12	5.1996	0.1920	0.1969	0.2272

Short audio did not show an e2e gain:

• Mean latency: +3.00%
• Throughput: -2.90%

This indicates the short-audio workload is not dominated enough by the encoder path.

E2E benchmark: long audio

Config:

• Audio: results/query_to_cars_120s.wav
• Per-request audio duration: 110.88s
• Total audio duration: 443.52s
• Requests: 4
• Concurrency: 1
• Warmup: 1

Mode	success	throughput req/s	latency mean s	latency p50 s	latency p95 s
encoder compile off	4/4	1.7085	0.5845	0.5975	0.7859
encoder compile on	4/4	1.8119	0.5509	0.5353	0.7997

Long audio showed a measurable e2e improvement:

• Mean latency: -5.75%
• Throughput: +6.05%
• Wall clock: -5.71%
• p50 latency: -10.41%
• p95 latency: +1.76%

The long-audio workload is more encoder-heavy, so encoder compile gives a visible e2e benefit.

Compile mode selection

Compile mode	Result	Decision
reduce-overhead	Faster in microbenchmark, but unstable in e2e with output drift and a cudagraph-related assertion	Rejected
max-autotune-no-cudagraphs, dynamic=False	Encoder-only speedup and long-audio e2e improvement	Selected
max-autotune-no-cudagraphs, dynamic=True	No clear advantage over static shapes and higher first-real-input compile/autotune cost	Rejected

Checklist

• Format your code according with pre-commit.
• Add unit tests.
• Update documentation / docstrings / example tutorials as needed.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.

CI

CI runs on self-hosted GPU runners and requires a maintainer to add the run-ci label. Once labeled, every subsequent push re-triggers CI as long as the label remains. Use /tag-and-rerun-ci higgs or /tag-and-rerun-ci moss to select a TTS CI model. Draft PRs are skipped even if labeled.