Commit 698a951
authored
support cuda_graph on iluvatar (#232)
### PR Category
Others
### PR Type
Bug Fixes
### Description
support cuda_graph on iluvatar,Has been verified Qwen3.6-27B and
Qwen3.6-35B-A3B models
test command:
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000
export VLLM_RPC_TIMEOUT=72000000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200
vllm serve /models/Qwen3.6-35B-A3B/ -tp 8 --served-model-name qwen
--port 8070 --max-model-len 131072 --compilation-config
'{"cudagraph_mode": "FULL_DECODE_ONLY"}'
Note: If use the default `compilation-config` parameter, maybe OOM.
### Related Issues
<!-- Link any related issues: Fixes #issue, Closes #issue, or Related to
#issue -->
### Changes
<!-- List the key changes made in this PR. -->
-
### Testing
<!-- How has this change been tested? Include test commands, hardware
used, etc. -->
-
### Checklist
- [ ] I have run the existing tests and they pass
- [ ] I have added tests for my changes (if applicable)
- [ ] I have updated the documentation (if applicable)1 parent 8728993 commit 698a951
1 file changed
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
333 | 333 | | |
334 | 334 | | |
335 | 335 | | |
336 | | - | |
| 336 | + | |
337 | 337 | | |
338 | 338 | | |
339 | 339 | | |
| |||
414 | 414 | | |
415 | 415 | | |
416 | 416 | | |
417 | | - | |
| 417 | + | |
418 | 418 | | |
419 | 419 | | |
420 | 420 | | |
| |||
0 commit comments