[hot-fix] 针对2.0.0 修复 flagrelease 发布中遇到的两个bug。 - #350
Closed
tonyh168 wants to merge 7 commits into
Closed
Conversation
# Description Add CUDA benchmark smoke coverage for vLLM bench in CI. This PR introduces a reusable benchmark workflow, extends the unified test runner with a benchmark scope, and adds pytest-based smoke tests for throughput, latency, and serve. The new benchmark stage runs after functional tests and validates benchmark entrypoints, command execution, and result outputs with lightweight CUDA benchmark cases. ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] Infra/Build change (changes to CI/CD workflows or build scripts) - [x] Code refactoring - [ ] Bug fix - [ ] Documentation change - [ ] Breaking change ## Changes - Add reusable CI workflow for benchmark smoke tests - Add benchmark stage to the platform test pipeline - Extend run.py with benchmark test discovery and execution - Add pytest smoke tests for throughput, latency, and serve - Add shared benchmark helpers for CLI argument generation and command execution - Add CUDA A100 benchmark smoke cases with local model/tokenizer paths ## Checklist - [x] I have read and followed the contributing guidelines - [x] The functionality is complete - [x] I have commented my code, particularly in CI workflow setup steps - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added/updated tests that prove my feature works on CUDA platform - [x] New and existing unit tests pass locally on CUDA platform --------- Co-authored-by: HermiaHuan <3081497279@qq.com> Co-authored-by: XMing <xmhubj@gmail.com>
### PR Category Tools ### PR Type Performance ### Description This benchmark script runs throughput/latency tests against a running vLLM serve endpoint and exports both raw and averaged metrics to CSV files. **How to use** 1. Start the vLLM server Use a command like: ```bash vllm serve /models/Qwen3.6-27B --tensor-parallel-size 2 --max-model-len 262144 --no-enable-log-requests --no-enable-prefix-caching ``` 2. Run benchmark with default cases ```bash python benchmarks/benchmark_throughput_serve.py ``` Each test case is: `(random_input_len, random_output_len, max_concurrency, num_prompts)` - `random_input_len`: prefill token length per request - `random_output_len`: generated token length per request - `max_concurrency`: max concurrent requests - `num_prompts`: total requests for that benchmark run Output files - `raw_runs_<timestamp>.csv`: one row per run - `summary_<timestamp>.csv`: averaged row per test case (after warmup skip) ``` # cat summary_20260522_094402.csv Prefill,Decode,Conc,Num Prompts,Benchmark Duration (s),Total Input Tokens,Total Output Tokens,Req/s,Output tok/s,Peak Output tok/s,Total tok/s,Mean TTFT (ms),Median TTFT (ms),P99 TTFT (ms),Mean TPOT (ms),Median TPOT (ms),P99 TPOT (ms),Mean ITL (ms),Median ITL (ms),P99 ITL (ms) 1024,1024,64,256,73.59,262144.0,262144.0,3.48,3562.08,4288.0,7124.16,940.97,981.23,2154.05,17.05,17.1,17.9,17.05,16.13,17.89 4096,1024,64,256,107.21,1048576.0,262144.0,2.39,2445.2,3840.0,12225.99,2117.0,1347.29,8819.76,24.08,25.08,25.93,24.08,17.99,269.88 ```
### Description Support for hygon adap, You must set the FlagCX path in bash before you use vllm-plugin-FL, such as: export FLAGCX_PATH=/workspace/FlagCX/ Note: This modification and test depends on FlagGems [PR #3477](flagos-ai/FlagGems#3477) ### Changes Add hygon device in VENDOR_DEVICE_MAP Add hygon.yaml in vllm_fl/dispatch/config ### Testing Run Qwen3.5-35B-A3B、Qwen3.6-35B-A3B、Qwen3.6-27B success。
### PR Category Core ### PR Type Bug Fixes ### Description Fix method miss when setting USE_FLAGGEMS=false. Block gems op index_put and nonzero to skip error while benchmark e2e case of 16k_1k.
### PR Category Vendor ### PR Type Bug Fixes ### Description Adapt for hygon ### Related Issues <!-- Link any related issues: Fixes #issue, Closes #issue, or Related to #issue --> ### Changes <!-- List the key changes made in this PR. --> - ### Testing <!-- How has this change been tested? Include test commands, hardware used, etc. --> - ### Checklist - [ ] I have run the existing tests and they pass - [ ] I have added tests for my changes (if applicable) - [ ] I have updated the documentation (if applicable) Signed-off-by: ftgreat <ldwang@baai.ac.cn>
GroupedTopKRouterFL._compute_routing called self._valid_grouping(), which does not exist on upstream GroupedTopKRouter (it uses an inner valid_grouping() closure), raising AttributeError during grouped top-k routing. Port the inline closure from v0.3.0-dev.
vllm 0.20.0 renamed the MLACommonImpl decode override from _forward_decode to forward_mqa (same signature). MLAFLImpl still defined _forward_decode, so its implementation was never dispatched and the abstract forward_mqa raised NotImplementedError at decode. Rename to match the base class.
|
loopyt seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
hotfix/v0.2.0: fix GroupedTopKRouterFL routing and MLA decode method name
Body(Markdown)
Summary
基于
v0.2.0tag 的 hotfix,修复两个运行时 bug。共两个 commit,各修一处。Bug 1 — GroupedTopKRouterFL 路由报错 (
4b6c1b2)vllm_fl/ops/fused_moe/router.py中GroupedTopKRouterFL._compute_routing调用了
self._valid_grouping(router_logits),但上游GroupedTopKRouter并没有
_valid_grouping方法 —— 它在_compute_routing内部用一个局部闭包valid_grouping()做校验。因此启用 grouped top-k 路由(DeepSeek-V2/V3、GLM-MoE、混元等模型)时会抛
AttributeError。修复:移植 v0.3.0-dev 的做法,用内联的
valid_grouping()闭包替换该调用(判断逻辑:
num_experts > num_expert_group且能被整除)。仅移植该最小修复,不含 v0.3.0-dev 中无关的
call_op→CachedOpdispatch 重构。```python
def valid_grouping() -> bool:
num_experts = router_logits.shape[-1]
if num_experts <= self.num_expert_group:
return False
return num_experts % self.num_expert_group == 0
if not valid_grouping():
...
```
Bug 2 — MLA decode 方法签名对不上 (
2bac207)vllm_fl/dispatch/backends/flaggems/impl/mla.py中MLAFLImpl继承自MLACommonImpl。vllm 0.20.0 把基类的 decode 覆写方法从_forward_decode重命名为
forward_mqa(参数签名完全相同)。FL 侧仍定义_forward_decode,导致该实现永远不会被调用,decode 时命中基类抽象的
forward_mqa→ 抛NotImplementedError。修复:将
MLAFLImpl._forward_decode重命名为forward_mqa,方法体不变。(metax 后端用的是自带 vendored 的 MLA,不受影响,未改动。)
Test
python -m py_compilerouter.py中不再存在self._valid_grouping调用mla.py中 decode 方法名为forward_mqa