Skip to content

Commit 4305d8e

Browse files
Mcirino1Mcirino1
authored
Updated README.md for June 24 Docker release
1 parent 0f7ec48 commit 4305d8e

1 file changed

Lines changed: 63 additions & 48 deletions

File tree

‎docs/dev-docker/README.md‎

Lines changed: 63 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ The pre-built image includes:
1212

1313
- ROCm™ 6.4.1
1414
- HipblasLT 0.15
15-
- vLLM 0.9.0.1
15+
- vLLM 0.9.1
1616
- PyTorch 2.7
1717

1818
## Pull latest Docker Image
@@ -21,14 +21,14 @@ Pull the most recent validated docker image with `docker pull rocm/vllm-dev:main
2121

2222
## What is New
2323

24-
- Updated to ROCm 6.4.1 and vLLM v0.9.0.1
25-
- AITER MHA
26-
- IBM 3d kernel for unified attention
27-
- Full graph capture for split attention
24+
- V1 on by default (use VLLM_USE_V1=0 to override)
25+
- Fixed detokenizers issue
26+
- Fixed AITER MoE issues
27+
- vLLM v0.9.1
2828

2929
## Known Issues and Workarounds
3030

31-
- No AITER MoE. Do not use VLLM_ROCM_USE_AITER for Mixtral or DeepSeek models.
31+
- --disable-custom-all-reduce required for Llama-3.1 405B
3232

3333
## Performance Results
3434

@@ -45,51 +45,53 @@ The table below shows performance data where a local inference client is fed req
4545
| | | | 128 | 4096 | 1500 | 1500 | 13667.3 |
4646
| | | | 500 | 2000 | 2000 | 2000 | 13367.1 |
4747
| | | | 2048 | 2048 | 1500 | 1500 | 8352.6 |
48-
| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 1500 | 1500 | 4275.0 |
49-
| | | | 128 | 4096 | 1500 | 1500 | 3356.7 |
50-
| | | | 500 | 2000 | 2000 | 2000 | 3201.4 |
51-
| | | | 2048 | 2048 | 500 | 500 | 2179.7 |
48+
| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 1500 | 1500 | 3329.1 |
49+
| | | | 128 | 4096 | 1500 | 1500 | 2733.0 |
50+
| | | | 500 | 2000 | 2000 | 2000 | 2765.0 |
51+
| | | | 2048 | 2048 | 500 | 500 | 2170.1 |
5252

5353
*TP stands for Tensor Parallelism.*
5454

55+
Supermicro AS-8125GS-TNMR2 with 2x AMD EPYC 9554 Processors, 2.25 TiB RAM, 8x AMD Instinct MI300X (192GiB, 750W) GPUs, Ubuntu 22.04, and amdgpu driver 6.8.5
56+
5557
### Latency Measurements
5658

5759
The table below shows latency measurement, which typically involves assessing the time from when the system receives an input to when the model produces a result.
5860

5961
| Model | Precision | TP Size | Batch Size | Input | Output | MI300X Latency (sec) |
6062
|-------|-----------|----------|------------|--------|---------|-------------------|
61-
| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 15.566 |
62-
| | | | 2 | 128 | 2048 | 16.858 |
63-
| | | | 4 | 128 | 2048 | 17.518 |
64-
| | | | 8 | 128 | 2048 | 18.898 |
65-
| | | | 16 | 128 | 2048 | 21.023 |
66-
| | | | 32 | 128 | 2048 | 23.896 |
67-
| | | | 64 | 128 | 2048 | 30.753 |
68-
| | | | 128 | 128 | 2048 | 43.767 |
69-
| | | | 1 | 2048 | 2048 | 15.496 |
70-
| | | | 2 | 2048 | 2048 | 17.380 |
71-
| | | | 4 | 2048 | 2048 | 17.983 |
72-
| | | | 8 | 2048 | 2048 | 19.771 |
73-
| | | | 16 | 2048 | 2048 | 22.702 |
74-
| | | | 32 | 2048 | 2048 | 27.392 |
75-
| | | | 64 | 2048 | 2048 | 36.879 |
76-
| | | | 128 | 2048 | 2048 | 57.003 |
77-
| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 45.828 |
78-
| | | | 2 | 128 | 2048 | 46.757 |
79-
| | | | 4 | 128 | 2048 | 48.322 |
80-
| | | | 8 | 128 | 2048 | 51.479 |
81-
| | | | 16 | 128 | 2048 | 54.861 |
82-
| | | | 32 | 128 | 2048 | 63.119 |
83-
| | | | 64 | 128 | 2048 | 82.362 |
84-
| | | | 128 | 128 | 2048 | 109.698 |
85-
| | | | 1 | 2048 | 2048 | 46.514 |
86-
| | | | 2 | 2048 | 2048 | 47.271 |
87-
| | | | 4 | 2048 | 2048 | 49.679 |
88-
| | | | 8 | 2048 | 2048 | 54.366 |
89-
| | | | 16 | 2048 | 2048 | 60.390 |
90-
| | | | 32 | 2048 | 2048 | 74.209 |
91-
| | | | 64 | 2048 | 2048 | 104.728 |
92-
| | | | 128 | 2048 | 2048 | 154.041 |
63+
| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 17.175 |
64+
| | | | 2 | 128 | 2048 | 17.603 |
65+
| | | | 4 | 128 | 2048 | 18.128 |
66+
| | | | 8 | 128 | 2048 | 19.549 |
67+
| | | | 16 | 128 | 2048 | 21.518 |
68+
| | | | 32 | 128 | 2048 | 24.103 |
69+
| | | | 64 | 128 | 2048 | 31.443 |
70+
| | | | 128 | 128 | 2048 | 42.932 |
71+
| | | | 1 | 2048 | 2048 | 17.112 |
72+
| | | | 2 | 2048 | 2048 | 17.857 |
73+
| | | | 4 | 2048 | 2048 | 18.711 |
74+
| | | | 8 | 2048 | 2048 | 19.770 |
75+
| | | | 16 | 2048 | 2048 | 21.865 |
76+
| | | | 32 | 2048 | 2048 | 25.302 |
77+
| | | | 64 | 2048 | 2048 | 33.435 |
78+
| | | | 128 | 2048 | 2048 | 48.935 |
79+
| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 52.201 |
80+
| | | | 2 | 128 | 2048 | 52.689 |
81+
| | | | 4 | 128 | 2048 | 53.543 |
82+
| | | | 8 | 128 | 2048 | 56.713 |
83+
| | | | 16 | 128 | 2048 | 62.190 |
84+
| | | | 32 | 128 | 2048 | 68.914 |
85+
| | | | 64 | 128 | 2048 | 85.783 |
86+
| | | | 128 | 128 | 2048 | 116.485 |
87+
| | | | 1 | 2048 | 2048 | 52.309 |
88+
| | | | 2 | 2048 | 2048 | 52.551 |
89+
| | | | 4 | 2048 | 2048 | 53.685 |
90+
| | | | 8 | 2048 | 2048 | 56.905 |
91+
| | | | 16 | 2048 | 2048 | 63.313 |
92+
| | | | 32 | 2048 | 2048 | 71.007 |
93+
| | | | 64 | 2048 | 2048 | 88.708 |
94+
| | | | 128 | 2048 | 2048 | 124.583 |
9395

9496
*TP stands for Tensor Parallelism.*
9597

@@ -200,6 +202,8 @@ Some environment variables enhance the performance of the vLLM kernels on the MI
200202

201203
```bash
202204
export VLLM_USE_TRITON_FLASH_ATTN=0
205+
export VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1
206+
203207
```
204208

205209
### vLLM engine performance settings
@@ -242,9 +246,12 @@ python3 /app/vllm/benchmarks/benchmark_latency.py \
242246
--num-iters-warmup 3 \
243247
--num-iters 5 \
244248
--output-json output.json
249+
--disable-custom-all-reduce
250+
--compilation-config '{"full_cuda_graph": true,"custom_ops":["+rms_norm","+silu_and_mul"],"pass_config":{"enable_noop":true,"enable_fusion":true}}’
251+
245252
```
246253
247-
For FP16 models, remove `--quantization fp8 --kv-cache-dtype fp8`.
254+
For FP16 models, remove `--quantization fp8 --kv-cache-dtype fp8`. For all other models, remove `--disable-custom-all-reduce`.
248255
249256
When measuring models with long context lengths, performance may improve by setting `--max-model-len` to a smaller value. It is important, however, to ensure that the `--max-model-len` is at least as large as the IN + OUT token counts.
250257
@@ -290,9 +297,11 @@ python3 /app/vllm/benchmarks/benchmark_throughput.py \
290297
--num-prompts $PROMPTS \
291298
--max-num-seqs $MAX_NUM_SEQS \
292299
--output-json output.json
300+
--disable-custom-all-reduce
301+
--compilation-config '{"full_cuda_graph": true,"custom_ops":["+silu_and_mul"],"pass_config":{"enable_noop":true,"enable_fusion":true}}’
293302
```
294303
295-
For FP16 models, remove `--quantization fp8 --kv-cache-dtype fp8`.
304+
For FP16 models, remove `--quantization fp8 --kv-cache-dtype fp8`. For all other models, remove `--disable-custom-all-reduce`.
296305
297306
When measuring models with long context lengths, performance may improve by setting `--max-model-len` to a smaller value (8192 in this example). It is important, however, to ensure that the `--max-model-len` is at least as large as the IN + OUT token counts.
298307
@@ -357,7 +366,7 @@ docker run -it --rm --ipc=host --network=host --group-add render \
357366
--cap-add=CAP_SYS_ADMIN --cap-add=SYS_PTRACE \
358367
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
359368
-e VLLM_USE_TRITON_FLASH_ATTN=1 \
360-
-e VLLM_USE_AITER=1 \
369+
-e VLLM_ROCM_USE_AITER=1 \
361370
-e VLLM_MLA_DISABLE=0 \
362371
rocm/vllm-dev:main
363372
@@ -447,14 +456,14 @@ You should see some performance improvement about the e2e latency.
447456
448457
### AITER use cases
449458
450-
`rocm/vllm-dev:main` image has experimental [AITER](https://github.com/ROCm/aiter) support, and can yield siginficant performance increase for some model/input/output/batch size configurations. To enable the feature make sure the following environment is set: `VLLM_USE_AITER=1`, the default value is `0`. When building your own image follow the [Docker build steps](#Docker-manifest) using the [aiter_integration_final](https://github.com/ROCm/vllm/tree/aiter_integration_final) branch.
459+
`rocm/vllm-dev:main` image has experimental [AITER](https://github.com/ROCm/aiter) support, and can yield siginficant performance increase for some model/input/output/batch size configurations. To enable the feature make sure the following environment is set: `VLLM_ROCM_USE_AITER=1`, the default value is `0`. When building your own image follow the [Docker build steps](#Docker-manifest) using the [aiter_integration_final](https://github.com/ROCm/vllm/tree/aiter_integration_final) branch.
451460
452461
Some use cases include:
453462
- amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV
454463
- amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV
455464
456465
```bash
457-
export VLLM_USE_AITER=1
466+
export VLLM_ROCM_USE_AITER=1
458467
python3 /app/vllm/benchmarks/benchmark_latency.py --model amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV -tp 8 --batch-size 256 --input-len 128 --output-len 2048
459468
```
460469
@@ -491,7 +500,7 @@ To reproduce the release docker:
491500
```bash
492501
git clone https://github.com/ROCm/vllm.git
493502
cd vllm
494-
git checkout 71faa188073d427c57862c45bf17745f3b54b1b1
503+
git checkout b335519f20495128a47d86f2c01dd467e2fe602b
495504
docker build -f docker/Dockerfile.rocm -t <your_tag> --build-arg USE_CYTHON=1 .
496505
```
497506
@@ -508,6 +517,12 @@ Use AITER release candidate branch instead:
508517
509518
## Changelog
510519
520+
20250620_aiter:
521+
- V1 on by default (use VLLM_USE_V1=0 to override)
522+
- Fixed detokenizers issue
523+
- Fixed AITER MoE issues
524+
- vLLM v0.9.1
525+
511526
20250605_aiter:
512527
- Updated to ROCm 6.4.1 and vLLM v0.9.0.1
513528
- AITER MHA

0 commit comments

Comments
 (0)