Skip to content

Commit 545561d

Browse files
Fixing pre-commit complaints.
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
1 parent 49efd27 commit 545561d

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

docs/dev-docker/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,7 @@ python3 /app/vllm/benchmarks/benchmark_throughput.py \
291291
--num-prompts $PROMPTS \
292292
--max-num-seqs $MAX_NUM_SEQS
293293
```
294+
294295
For FP16 models, remove `--kv-cache-dtype fp8`.
295296

296297
When measuring models with long context lengths, performance may improve by setting `--max-model-len` to a smaller value (8192 in this example). It is important, however, to ensure that the `--max-model-len` is at least as large as the IN + OUT token counts.
@@ -325,6 +326,7 @@ vllm serve amd/Llama-3.1-70B-Instruct-FP8-KV \
325326
--gpu-memory-utilization 0.99 \
326327
--num_scheduler-steps 10
327328
```
329+
328330
For FP16 models, remove `--kv-cache-dtype fp8`. Change port (for example --port 8005) if port=8000 is currently being used by other processes.
329331

330332
Run client in a separate terminal. Use port_id from previous step else port-id=8000.

0 commit comments

Comments
 (0)