Skip to content

Commit 698a951

Browse files
authored
support cuda_graph on iluvatar (#232)
### PR Category Others ### PR Type Bug Fixes ### Description support cuda_graph on iluvatar,Has been verified Qwen3.6-27B and Qwen3.6-35B-A3B models test command: export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000 export VLLM_RPC_TIMEOUT=72000000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 vllm serve /models/Qwen3.6-35B-A3B/ -tp 8 --served-model-name qwen --port 8070 --max-model-len 131072 --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' Note: If use the default `compilation-config` parameter, maybe OOM. ### Related Issues <!-- Link any related issues: Fixes #issue, Closes #issue, or Related to #issue --> ### Changes <!-- List the key changes made in this PR. --> - ### Testing <!-- How has this change been tested? Include test commands, hardware used, etc. --> - ### Checklist - [ ] I have run the existing tests and they pass - [ ] I have added tests for my changes (if applicable) - [ ] I have updated the documentation (if applicable)
1 parent 8728993 commit 698a951

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

vllm_fl/platform.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -333,7 +333,7 @@ def get_static_graph_wrapper_cls(cls) -> str:
333333

334334
@classmethod
335335
def support_static_graph_mode(cls) -> bool:
336-
if cls.vendor_name in ["nvidia", "ascend", "metax", "hygon", "mthreads"]:
336+
if cls.vendor_name in ["nvidia", "ascend", "metax", "hygon", "mthreads", "iluvatar"]:
337337
return True
338338
return False
339339

@@ -414,7 +414,7 @@ def get_device_total_memory(cls, device_id: int = 0) -> int:
414414

415415
@classmethod
416416
def use_custom_op_collectives(cls) -> bool:
417-
return cls.vendor_name in ("nvidia", "thead")
417+
return cls.vendor_name in ("nvidia", "thead", "iluvatar")
418418

419419
@classmethod
420420
def num_compute_units(cls, device_id: int = 0) -> int:

0 commit comments

Comments
 (0)