backend_comparison_note.md

Inference Optimization in vLLM

1. Memory Management & Caching

The foundation of vLLM performance is efficient VRAM memory management, which allows handling a larger number of simultaneous requests (larger batch size).

Functionality	Description	Flags / Configuration
PagedAttention	(Core) Splits KV Cache into fixed blocks (pages), eliminating memory fragmentation. Works analogously to virtual memory in OSs. Allows for near-zero VRAM waste.	--block-size <int> (default: 16) – page block size.
Automatic Prefix Caching (APC)	Detects repeating prompt fragments (e.g., System Prompt, chat history) and stores their KV Cache. Prevents re-computation of the same tokens.	--enable-prefix-caching (default: False) Requires manual enabling.
Context Limiting	Limiting the maximum sequence length saves memory on KV Cache, increasing available batch size.	--max-model-len <int> (e.g., 8192) – if not set, vLLM tries to allocate as much as the model config specifies (often risky for OOM).
GPU Utilization	Determines how much GPU memory vLLM reserves at start (mainly for KV Cache). Higher value = higher throughput.	--gpu-memory-utilization <float> (default: 0.9).
Swap Space	CPU RAM buffer in case of VRAM overflow. Prevents OOM errors at the cost of temporary performance drops (GPU<->CPU page swapping).	--swap-space <GiB> (default: 4).

2. Batching & Scheduling

Mechanisms deciding how requests are queued and processed by the GPU.

Functionality	Description	Flags / Configuration
Continuous Batching	(Core) Processing at the iteration (token) level, not the entire request. When one request in a batch finishes, a new one enters immediately without waiting for the rest.	Built-in, works automatically. Load control via: --max-num-seqs <int> (default: 256).
Chunked Prefill	Splits the "prefill" phase (processing a long prompt) into smaller chunks, interleaving them with generation (decode) of other requests. Prevents system blocking by one long prompt and allows better queue management.	--enable-chunked-prefill (default: False). --max-num-batched-tokens <int> – defines chunk size (token budget). Advanced queue control: --max-num-partial-prefills <int> (default: 1) – maximum number of sequences that can be processed in "partial prefill" mode simultaneously. --max-long-partial-prefills <int> (default: 1) – limit of parallel prompts considered "long". Setting a value lower than max-num-partial-prefills allows shorter prompts to "skip" the queue before long ones, improving latency for simple requests. --long-prefill-token-threshold <int> (default: 0) – token count threshold above which a prompt is classified as "long" (for the flag above).
Async Output Processing	Moves operations like detokenization and sampling to a separate thread/process to avoid blocking the main GPU loop.	Built-in in newer versions (v0.6+).
Dual Batch Overlap (DBO)	(Advanced) Technique involving splitting the batch into smaller micro-batches to parallelize different processing stages (e.g., overlapping CPU work with GPU). Increases hardware utilization under heavy load.	--enable-dbo (default: False) – Enables Dual Batch Overlap mechanism. --ubatch-size <int> (default: 0) – Defines micro-batch size. Activation Thresholds: The system uses micro-batching only when the token count exceeds a certain threshold (to avoid overhead on small requests): --dbo-decode-token-threshold <int> (default: 32) – threshold for decode phase. --dbo-prefill-token-threshold <int> (default: 512) – threshold for prefill phase.

3. Parallelism & Scaling

Methods allowing handling of models exceeding the capabilities of a single GPU card or a single server.

Functionality	Description	Flags / Configuration
Tensor Parallelism (TP)	Splits model weights of individual layers across multiple GPUs (intra-layer). Requires fast interconnect (NVLink). Default method for large models on a single node.	--tensor-parallel-size <int> or -tp.
Pipeline Parallelism (PP)	Splits model into groups of layers distributed across different GPUs (inter-layer). Lower performance than TP, but works better on slower interconnects (multi-node).	--pipeline-parallel-size <int> or -pp. --distributed-executor-backend ray
Expert Parallelism (EP)	Specific to MoE models (e.g., Mixtral). Distributes "experts" across different GPUs.	--enable-expert-parallelism (usually auto-detected with TP).
Context Parallelism (CP)	Splits sequence dimension (KV Cache) across multiple GPUs. Used for extremely long contexts (e.g., 100k+). Allows independent parallelism control for prompt processing and generation phases.	Phase Configuration: --decode-context-parallel-size <int> or -dcp (default: 1) – Number of CP groups for decode phase. Does not change total GPU count (world size), but utilizes GPUs assigned to TP. Requirement: tp_size must be divisible by dcp_size. --prefill-context-parallel-size <int> or -pcp (default: 1) – Number of CP groups for prefill phase (prompt processing).
Data Parallelism (DP)	Runs multiple model replicas to increase throughput. In MoE models, it also affects how experts are sharded.	--data-parallel-size <int> or -dp (default: 1).

4. Compute & Latency Optimization

Techniques accelerating mathematical operations or reducing driver overhead.

Functionality	Description	Flags / Configuration
CUDA Graphs	"Records" GPU operation sequence to run them with a single call. Reduces CPU overhead with small batches.	Enabled by default. --enforce-eager (setting to True disables this optimization). --disable-custom-all-reduce (might be needed with TP issues).
Speculative Decoding	Uses a small model (draft model) to propose tokens that the large model only verifies. Reduces single request latency.	--speculative-model <path> --num-speculative-tokens <int>
Quantization	Support for compressed models (FP8, AWQ, GPTQ, SqueezeLLM). Accelerates computation and reduces memory usage.	--quantization <method> or -q (e.g., awq, fp8). If None, loads from model config.
Fused Kernels	(v0.13+) Optimized kernels combining multiple operations (e.g., norms, activations) into a single memory pass.	Built-in, run automatically for supported architectures.

5. Architecture Features

Advanced deployment scenarios.

Functionality	Description	Flags / Configuration
Multi-LoRA Serving	Support for multiple LoRA adapters on a single base model. Adapters are loaded dynamically per request.	--enable-lora --max-loras <int> --max-lora-rank <int>
Disaggregated Prefill	Architecture separating "Prefill" and "Decode" instances. Requires fast KV Cache transfer (Mooncake/Nixl connectors).	Configuration at orchestration level (not a single flag, but system architecture). Uses --kv-transfer-config.
Encoder Batching	(v0.13+, for Whisper models) Enables parallel encoder phase processing for audio models.	Automatic in newer versions for supported models.

___________________________________

Triton + TensorRT-LLM Optimization

1. Memory Management & KV Cache

Efficient VRAM management for keys and values (KV) is crucial for increasing Batch Size.

Functionality	Description	Flags / Configuration
Paged KV Cache	(Core) Splits cache into blocks (pages), eliminating fragmentation. Allows dynamic memory allocation for growing sequences.	Runtime (tensorrt_llm): kv_cache_free_gpu_mem_fraction (default: 0.9) – Key parameter. Determines how much free GPU memory (after loading model) to dedicate to cache. Higher value = larger batch. max_tokens_in_paged_kv_cache – Optional "hard" limit of tokens in cache.
KV Cache Reuse (Prefix Caching)	Allows reuse of memory blocks for repeating prefixes (e.g., System Prompt). Essential for RAG and multi-turn chats.	Runtime: enable_kv_cache_reuse: "true" (default: false).
Host Offloading	Moving part of KV Cache to RAM (CPU) when VRAM is lacking. Prevents OOM errors at the cost of performance.	Runtime: kv_cache_host_memory_bytes – Amount of RAM dedicated to offload.
KV Cache Quantization	Storing KV Cache in INT8 or FP8 format instead of FP16. Reduces memory usage by half, potentially doubling batch size.	Build/Convert: int8_kv_cache: true kv_cache_type (continuous/paged).

2. Scheduling & Batching

Mechanisms determining throughput and latency.

Functionality	Description	Flags / Configuration
Inflight Batching (IFB)	(Core) Equivalent to Continuous Batching. Dynamically adds and removes requests from GPU during generation without waiting for the whole batch to finish.	Runtime: batching_strategy: "inflight_fused_batching" – This setting is critical for performance. "V1" value (traditional) is significantly slower.
Scheduler Policy	Request packing strategy. Do we risk pausing requests to handle more ("greedy"), or guarantee no interruptions?	Runtime: batch_scheduler_policy - guaranteed_no_evict (Default): Stable latency, no pause risk. - max_utilization: Aggressive packing, higher throughput, risk of pausing generation upon memory shortage.
Queue Optimization	Artificial delay in processing start to collect a larger group of requests (input micro-batching).	Runtime: max_queue_delay_microseconds – Wait time to complete a batch. Setting > 0 may increase throughput at the cost of first token latency.

3. Parallelism

Configured mainly at the engine build stage (convert/build), defines distributed architecture.

Functionality	Description	Flags / Configuration
Tensor Parallelism (TP)	Splitting model weights inside layers across multiple GPUs. Requires fast interconnect (NVLink).	Build/Convert: tp_size (e.g., 1, 2, 4, 8).
Pipeline Parallelism (PP)	Splitting model into layer groups between GPUs/Nodes.	Build/Convert: pp_size.
Context Parallelism (CP)	Splitting sequence dimension (Context) across multiple GPUs. For very long prompts.	Build/Convert: cp_size.
MoE Parallelism (EP)	Specific to Mixture-of-Experts. Defines expert splitting.	Build/Convert: moe_tp_size (TP for MoE). moe_ep_size (Expert Parallelism).
Auto Parallel	Automatic selection of parallelism strategy based on the cluster.	Build: auto_parallel: 1 gpus_per_node, cluster_key.

4. Compute & Kernels Optimization

Low-level optimizations that TensorRT is known for.

Functionality	Description	Flags / Configuration
Low Precision (Quantization)	Use of INT4/INT8 or FP8 weights. Significantly accelerates computation and reduces model size.	Build/Convert: use_weight_only: true weight_only_precision: 'int8' / 'int4_gptq' smoothquant (for activations). fp8_rowwise_gemm_plugin (for FP8 on H100).
Plugins	Specialized Nvidia kernel implementations (e.g., FlashAttention). Replace standard operations with faster equivalents.	Build: gpt_attention_plugin (auto/enable). gemm_plugin (auto/fp8). context_fmha (Fused Multi-Head Attention). remove_input_padding (Removing padding for performance).
CUDA Graphs	Recording GPU operation graph to reduce CPU overhead.	Runtime: cuda_graph_mode: "true" cuda_graph_cache_size – Number of graphs in cache (for different batch sizes).
Context Chunking	Splitting prompt processing phase into chunks to avoid blocking GPU with long texts (equivalent to Chunked Prefill).	Runtime: enable_chunked_context: "true"

5. Decoding & Features

Text generation methods and additional features.

Functionality	Description	Flags / Configuration
Speculative Decoding	Using "draft model" or other methods (Medusa, Lookahead) to accelerate generation.	Runtime: decoding_mode (e.g., medusa, lookahead). speculative_decoding_mode (Build).
Guided Decoding	Enforcing output format (e.g., JSON) according to a schema. C++ backend (xgrammar) is very fast.	Runtime: guided_decoding_backend: "xgrammar".
Decoupled Mode	Separating the receiving thread from the generating thread. Essential for streaming tokens.	Runtime: decoupled_mode: "True".
LoRA Support	Support for LoRA adapters. Requires enabling plugin during build.	Build: lora_plugin. Runtime: lora_cache_gpu_memory_fraction.

Summary - What to set in Triton Config (config.pbtxt)?

For a typical production deployment (High Performance):

1. batching_strategy: Must be set to "inflight_fused_batching".
2. kv_cache_free_gpu_mem_fraction: Set high (0.9 or 0.95) to maximize Batch Size.
3. enable_kv_cache_reuse: Enable ("true") if you have a RAG system or Chat.
4. decoupled_mode: Enable ("true") if the client application requires streaming responses.
5. cuda_graph_mode: Worth enabling ("true") to reduce CPU load.
6. enable_chunked_context: Enable ("true") if you handle very diverse prompt lengths (mix of short and very long).

Optimization Methods Comparison (vLLM vs Triton + TRT-LLM)

1. Architectural Foundation

• vLLM (JIT / Runtime Optimization):

  • Based on Pytorch.
  • Optimizations are applied "on the fly" (Just-In-Time) or via resource pre-allocation during server startup.
  • Kernel (PagedAttention) is a custom CUDA extension called from Python.
  • Computational graphs (CUDA Graphs) are "recorded" during the warmup phase.

• Triton + TensorRT-LLM (AOT / Static Optimization):

  • Based on a compiled engine (.engine) built by TensorRT.
  • Optimizations (layer fusion, kernel selection, precision) are "baked in" permanently during the compilation process (trtllm-build).
  • Triton acts as an orchestrator managing memory and queuing for the static C++ engine.

2. Functionality Comparison Table

The table below maps corresponding techniques in both solutions.

Category	Functionality	Implementation in vLLM	Implementation in Triton (TRT-LLM)
Memory	Paged KV Cache	PagedAttention (Native)	Paged KV Cache (Plugin/Native)
Memory	Prefix Caching	Automatic Prefix Caching (Runtime flag)	KV Cache Reuse (Runtime config)
Batching	Continuous Batching	Iteration-level Scheduling	Inflight Fused Batching (IFB)
Precision	Quantization	Loading AWQ/GPTQ/FP8 weights or on-the-fly conversion.	Compiling weights to INT8/FP8 engine (requires calibration or use_weight_only flags).
Context	Split Prefill	Chunked Prefill	Chunked Context
Speculation	Acceleration	Speculative Decoding (Draft Model loaded separately)	Speculative Decoding (Special engine or Medusa heads built into engine)
Kernels	Operation Optimization	Custom Pytorch Kernels + CUDA Graphs	Kernel Fusion (Horizontal/Vertical fusion) via TensorRT compiler
Output	Structured Output	outlines / xgrammar library	C++ backend xgrammar (Guided Decoding)

3. Detailed Technical Difference Analysis

A. Parallelism Management

This is the main point of technical divergence. Both systems support TP (Tensor), PP (Pipeline), EP (Expert), and CP (Context), but manage them differently.

• vLLM (Dynamic):

  • Parallelism is defined at startup (vllm serve --tensor-parallel-size 4).
  • The engine dynamically builds communication groups (NCCL) during initialization.
  • Technical advantage: Ability to change topology (e.g., from 2 to 4 GPUs) without changing model files.

• TensorRT-LLM (Static/Compiled):

  • Parallelism is defined during engine building (trtllm-build --tp_size 4).
  • Model weights are physically divided (sharding) and saved in separate files for each GPU before running.
  • Technical advantage: Compiler can optimize communication between GPUs (e.g., insert reduction instructions directly into compute kernels), eliminating dynamic planning overhead.

B. Compute Optimization & Kernels

• vLLM (CUDA Graphs & PagedAttention):

  • The main optimization is eliminating CPU overhead via CUDA Graphs. vLLM records operation sequences for various batch sizes and replays them.
  • Uses highly optimized kernels for Attention, but the rest of the model is often standard Pytorch operations (or optimized vllm-ops kernels).

• TensorRT-LLM (Kernel Fusion):

  • Utilizes Kernel Fusion at the compiler level. For example, MatMul + Bias + Activation operations are combined into a single GPU kernel, reducing VRAM memory accesses.
  • For Hopper architecture (H100), it generates specific kernels utilizing FP8 instructions and Transformer Engine in a way not standardly available in Pytorch.

C. LoRA (Low-Rank Adaptation) Support

• vLLM:

  • Treats LoRA adapters as additional tensors loaded into memory alongside the main model.
  • Uses special kernels (S-LoRA / Punica) that allow applying different adapters for different requests in the same batch without recompilation.

• Triton + TRT-LLM:

  • Requires enabling lora_plugin during engine building.
  • Reserves a static memory block for adapters (lora_cache_gpu_memory_fraction in Triton config).
  • Mechanism is more rigid (requires pre-allocation) but benefits from TRT kernel fusion for LoRA operations.

D. Medusa and Speculative Decoding

• vLLM:

  • Implementation is flexible: "Draft Model" is simply another model running in vLLM communicating with the main one.
  • For Medusa: loads Medusa head weights as an addition to the model.

• Triton + TRT-LLM:

  • Implementation is integrated: Speculation logic (token verification) is part of the TensorRT graph.
  • Requires defining medusa_choices (decision tree) in Triton configuration, allowing very fast verification in C++ without Python interpreter overhead.

E. Scheduling

• vLLM:

  • Scheduler runs in Python (though optimized). Decides which memory blocks to allocate and which requests enter the next model step.

• Triton (Inflight Batcher):

  • Scheduler is implemented in C++ as part of the backend.
  • Uses guaranteed_no_evict or max_utilization strategies.
  • Supports Decoupled Mode – separating the network thread (receiving requests) from the inference thread, which is critical for stable streaming under very heavy load.