README.md

Speculative Decoding with TensorRT-LLM Backend

Overview

TensorRT-LLM backend provides highly optimized speculative decoding support for NVIDIA GPUs. This guide covers configuration and deployment of speculative decoding using the TensorRT-LLM backend.

Documentation

For comprehensive documentation on speculative decoding with TensorRT-LLM, please refer to the official TensorRT-LLM backend documentation:

TensorRT-LLM Backend Decoding Documentation

Key Features

• Maximum performance on NVIDIA GPUs with optimized kernels
• Support for INT8/FP8 quantization
• Advanced scheduling and batching
• Medusa and standard speculative decoding modes

Quick Reference

For speculative decoding setup with TensorRT-LLM:

1. Build TensorRT-LLM engines for both target and draft models
2. Configure the model repository with appropriate parameters
3. Deploy using Triton with TensorRT-LLM backend

See the TensorRT-LLM backend documentation for detailed instructions.

See Also

• Speculative Decoding Overview
• vLLM Speculative Decoding (alternative backend)
• TensorRT-LLM Backend