Skip to content

docs: Update Llava guide to PyTorch backend with Qwen2.5-VL #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
faradawn wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Update Llava guide to PyTorch backend with Qwen2.5-VL #160

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 57 additions & 152 deletions Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,157 +26,74 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-->

# Deploying Hugging Face Llava1.5-7b Model in Triton
# Deploying a Vision-Language Model (Qwen2.5-VL) in Triton with TensorRT-LLM

TensorRT-LLM is Nvidia's recommended solution of running Large Language
Models(LLMs) on Nvidia GPUs. Read more about TensoRT-LLM [here](https://github.com/NVIDIA/TensorRT-LLM)
and Triton's TensorRT-LLM Backend [here](https://github.com/triton-inference-server/tensorrtllm_backend).
TensorRT-LLM is NVIDIA's recommended solution for running Large Language Models
(LLMs) and multimodal models on NVIDIA GPUs. Read more about TensorRT-LLM
[here](https://github.com/NVIDIA/TensorRT-LLM) and Triton's TensorRT-LLM Backend
[here](https://github.com/triton-inference-server/tensorrtllm_backend).

*NOTE:* If some parts of this tutorial doesn't work, it is possible that there
are some version mismatches between the `tutorials` and `tensorrtllm_backend`
repository. Refer to [llama.md](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama.md)
for more detailed modifications if necessary. And if you are familiar with
python, you can also try using
[LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md)
for LLM workflow.
This tutorial shows how to serve the multimodal
[Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
vision-language model with Triton Inference Server using the TensorRT-LLM
PyTorch backend (LLM API). The PyTorch backend works directly with Hugging Face
checkpoints — no TensorRT engine building required.

> [!NOTE]
> The legacy TensorRT engine-build workflow for multimodal models (building
> separate visual and LLM engines with `trtllm-build` and
> `build_visual_engine.py`, then wiring them through an `inflight_batcher_llm`
> model repository) is deprecated and is being removed from TensorRT-LLM. This
> tutorial uses the modern LLM API / PyTorch backend instead.

## Acquiring Llava1.5-7B model
## Launch the Triton TensorRT-LLM container

For this tutorial, we are using the Llava1.5-7B HuggingFace model with pre-trained
weights. Clone the repo of the model with weights and tokens
[here](https://huggingface.co/llava-hf/llava-1.5-7b-hf/tree/main).

## Deploying with Triton Inference Server

Next steps will guide you over the process of TensorRT and TensorRT-LLM engine
building and Triton model repository set up.

### Prerequisite: TensorRT-LLM backend

This tutorial requires TensorRT-LLM Backend repository. Please note,
that for best user experience we recommend using the latest
[release tag](https://github.com/triton-inference-server/tensorrtllm_backend/tags)
of `tensorrtllm_backend` and
the latest [Triton Server container.](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags)

To clone TensorRT-LLM Backend repository, make sure to run the following
set of commands.
```bash
git clone https://github.com/triton-inference-server/tensorrtllm_backend.git --branch <release branch>
# Update the submodules
cd tensorrtllm_backend
# Install git-lfs if needed
apt-get update && apt-get install git-lfs -y --no-install-recommends
git lfs install
git submodule update --init --recursive
```

### Launch Triton TensorRT-LLM container

Launch Triton docker container with TensorRT-LLM backend.
Note that we're mounting `tensorrtllm_backend` to `/tensorrtllm_backend`
and the Llava1.5 model to `/Llava-1.5-7b-hf` in the docker container for simplicity.
Make an `engines` folder outside docker to reuse engines for future runs.
Please, make sure to replace <xx.yy> with the version of Triton that you want
to use.
Mount your Hugging Face cache so the model can be auto-downloaded at server
startup. Replace `<xx.yy>` with the version of Triton you want to use — the
latest Triton Server container is recommended and can be found
[here](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags).

```bash
docker run --rm -it --net host --shm-size=2g \
--ulimit memlock=-1 --ulimit stack=67108864 --gpus all \
-v </path/to/tensorrtllm_backend>:/tensorrtllm_backend \
-v </path/to/Llava1.5/repo>:/llava-1.5-7b-hf \
-v </path/to/engines>:/engines \
-v </path/to/tutorials>:/tutorials \
-v ~/.cache/huggingface:/root/.cache/huggingface \
nvcr.io/nvidia/tritonserver:<xx.yy>-trtllm-python-py3
```

Alternatively, you can follow instructions
[here](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/build.md#build-the-docker-container)
to build Triton Server with Tensorrt-LLM Backend if you want
to build a specialized container.

Don't forget to allow gpu usage when you launch the container.

### Create Engines for each model [skip this step if you already have engines]

TensorRT-LLM requires each model to be compiled for the configuration
you need before running. To do so, before you run your model for the first time
on Triton Server you will need to create a TensorRT-LLM engine.
For gated models, set your token first: `export HF_TOKEN=hf_...`

Starting with [24.04 release](https://github.com/triton-inference-server/server/releases/tag/v2.45.0),
Triton Server TensrRT-LLM container comes with
pre-installed TensorRT-LLM package, which allows users to build engines inside
the Triton container.

Llava1.5 requires 2 engines: a TensorRT engine for visual components,
and a TRT-LLM engine for the language components. This tutorial bases on 24.05
release, which corresponds to `v0.9.0` version of TensorRT-LLM and
TensorRT-LLM backend and follows [this](https://github.com/NVIDIA/TensorRT-LLM/tree/v0.9.0/examples/multimodal#llava-and-vila)
TensorRT-LLM multi-modal guide.

To generate engines, simply follow the next steps:
## Prepare the model repository

```bash
HF_LLAVA_MODEL=/llava-1.5-7b-hf
UNIFIED_CKPT_PATH=/tmp/ckpt/llava/7b/
ENGINE_DIR=/engines/llava1.5
CONVERT_CHKPT_SCRIPT=/tensorrtllm_backend/tensorrt_llm/examples/llama/convert_checkpoint.py
python3 ${CONVERT_CHKPT_SCRIPT} --model_dir ${HF_LLAVA_MODEL} --output_dir ${UNIFIED_CKPT_PATH} --dtype float16
trtllm-build --checkpoint_dir ${UNIFIED_CKPT_PATH} \
--output_dir ${ENGINE_DIR} \
--gemm_plugin float16 \
--use_fused_mlp \
--max_batch_size 1 \
--max_input_len 2048 \
--max_output_len 512 \
--max_multimodal_len 576 # 1 (max_batch_size) * 576 (num_visual_features)

python /tensorrtllm_backend/tensorrt_llm/examples/multimodal/build_visual_engine.py --model_path ${HF_LLAVA_MODEL} --model_type llava --output_dir ${ENGINE_DIR}
git clone https://github.com/NVIDIA/TensorRT-LLM.git
```
Comment on lines 67 to 69

@greptile-apps greptile-apps Bot Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unversioned git clone risks reproducibility

Cloning from main without a branch or tag pin means the directory layout (triton_backend/all_models/llmapi/, triton_backend/scripts/launch_triton_server.py) and the model.yaml schema could change at any commit, silently breaking the tutorial. Adding --branch <release-tag> (e.g. the same tag that matches the <xx.yy> container version) would make the guide self-consistent and reproducible.


Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml`
and set the model:

> Optional: You can check test the output of the model with `run.py`
> located in the same llama examples folder.
>
> ```bash
> python3 /tensorrtllm_backend/tensorrt_llm/examples/multimodal/run.py --max_new_tokens 30 --hf_model_dir ${HF_LLAVA_MODEL} --visual_engine_dir ${ENGINE_DIR} --llm_engine_dir ${ENGINE_DIR} --decoder_llm --input_text "Question: which city is this? Answer:"
> ```
> You should expect the following response:
> ```
> [TensorRT-LLM] TensorRT-LLM version: 0.9.0
> ...
> [06/18/2024-01:02:24] [TRT-LLM] [I] ---------------------------------------------------------
> [06/18/2024-01:02:24] [TRT-LLM] [I]
> [Q] Question: which city is this? Answer:
> [06/18/2024-01:02:24] [TRT-LLM] [I]
> [A] ['Singapore']
> [06/18/2024-01:02:24] [TRT-LLM] [I] Generated 1 tokens
> [06/18/2024-01:02:24] [TRT-LLM] [I] ---------------------------------------------------------
> ```

### Serving with Triton
```yaml
model: Qwen/Qwen2.5-VL-7B-Instruct
backend: pytorch
```

The last step is to set up a Triton model repository. For this tutorial,
we provide all necessary Triton related files under `model_repository/`.
You simply need to provide TensorRT-LLM engine location in its `config.pbtxt`:
All keys in `model.yaml` map directly to the
[`LLM()` constructor arguments](https://nvidia.github.io/TensorRT-LLM/llm-api/) —
this is where you configure KV cache, parallelism, and more. You can also point
`model` at a local filesystem path if you have pre-downloaded the checkpoint.

```bash
FILL_TEMPLATE_SCRIPT=/tensorrtllm_backend/tools/fill_template.py
python3 ${FILL_TEMPLATE_SCRIPT} -i /tutorials/Popular_Models_Guide/Llava1.5/model_repository/tensorrt_llm/config.pbtxt engine_dir:${ENGINE_DIR}
```
## Serving with Triton

3. Launch Tritonserver
Launch Triton Server with the
[launch_triton_server.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/triton_backend/scripts/launch_triton_server.py)
script, running from the parent of `TensorRT-LLM/`:

Use the [launch_triton_server.py](https://github.com/triton-inference-server/tensorrtllm_backend/blob/release/0.5.0/scripts/launch_triton_server.py) script. This launches multiple instances of `tritonserver` with MPI.
```bash
export TRT_ENGINE_LOCATION="/engines/llava1.5/visual_encoder.engine"
export HF_LOCATION="/llava-1.5-7b-hf"
python3 /tensorrtllm_backend/scripts/launch_triton_server.py --world_size=<world size of the engine> --model_repo=/tutorials/Popular_Models_Guide/Llava1.5/model_repository
python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \
--model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/
```
Comment on lines 56 to 93

@greptile-apps greptile-apps Bot Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Missing volume mount breaks the workflow

The docker run command mounts only ~/.cache/huggingface, but the subsequent steps require access to the cloned TensorRT-LLM/ directory. If users follow the natural reading (clone on the host, then run Triton inside the container), the TensorRT-LLM/ path will not exist inside the container and the launch_triton_server.py command will fail. If the intent is to clone inside the running container, the guide should say so explicitly — but the --rm flag means the clone is discarded on container exit, making the workflow non-reproducible across sessions. A -v /path/to/TensorRT-LLM:/TensorRT-LLM mount (matching the git clone location on the host) needs to be added to the docker run command, or the guide must explicitly state that steps 2–4 are all performed inside a single interactive container session.

> You should expect the following response:

> You should expect the following response once the server is ready:
> ```
> ...
> I0503 22:01:25.210518 1175 grpc_server.cc:2463] Started GRPCInferenceService at 0.0.0.0:8001
> I0503 22:01:25.211612 1175 http_server.cc:4692] Started HTTPService at 0.0.0.0:8000
> I0503 22:01:25.254914 1175 http_server.cc:362] Started Metrics Service at 0.0.0.0:8002
Expand All @@ -187,37 +104,25 @@ To stop Triton Server inside the container, run:
pkill tritonserver
```

### Send an inference request
## Send an inference request

You can test the results of the run with:
1. The [multi_modal_client.py](./multi_modal_client.py) script.
For a text-only prompt, use the
[generate endpoint](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/protocol/extension_generate.html):

```bash
# Using the SDK container as an example
docker run --rm -it --net host --shm-size=2g \
--ulimit memlock=-1 --ulimit stack=67108864 --gpus all \
-v /path/to/tutorials:/tutorials
nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk

CLIENT_SCRIPT=/tutorials/Popular_Models_Guide/Llava1.5/multi_modal_client.py
python3 ${CLIENT_SCRIPT} --prompt "Describe the picture." --image_url "http://images.cocodataset.org/test2017/000000155781.jpg" --max-tokens=15
curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \
-d '{"text_input": "Describe how vision-language models understand images.", "sampling_param_max_tokens": 100}' | jq
```
> You should expect the following response:
> ```
> Got completed request
> The image features a city bus parked on the side of a street.
> ```

2. The [generate endpoint](https://github.com/triton-inference-server/tensorrtllm_backend/tree/release/0.5.0#query-the-server-with-the-triton-generate-endpoint).

```bash
curl -X POST localhost:8000/v2/models/llava-1.5/generate -d '{"prompt":"USER: <image>\nQuestion:Describe the picture. Answer:", "image":"http://images.cocodataset.org/test2017/000000155781.jpg", "max_tokens":100}'
```
> You should expect the following response:
> ```
> data: {"completion_tokens":77,"finish_reason":"stop","model_name":"llava-1.5","model_version":"1","prompt_tokens":592,"text":"The image features a city bus parked on the side of a street. The bus is positioned near a railroad crossing, and there is a stop sign visible in the scene. The bus is also displaying an \"Out of Service\" sign, indicating that it is not currently in operation. The street appears to be foggy, adding a sense of atmosphere to the scene.</s>","total_tokens":669}
> ```
For image + text (multimodal) requests, Qwen2.5-VL follows the TensorRT-LLM
multimodal LLM API input format. See the
[multimodal LLM API examples](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/models/core/multimodal/README.md)
and the
[TensorRT-LLM Backend LLM API guide](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llmapi.md)
for the exact request schema for passing images alongside the prompt.
Comment thread
greptile-apps[bot] marked this conversation as resolved.

## References

For more examples feel free to refer to [End to end workflow to run multi-modal models.](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/models/core/multimodal/README.md)
- [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md)
- [TensorRT-LLM Backend LLM API guide](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llmapi.md)
- [End to end workflow to run multi-modal models](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/models/core/multimodal/README.md)