Skip to content

[hot-fix] 针对2.0.0 修复 flagrelease 发布中遇到的两个bug。 - #350

Closed
tonyh168 wants to merge 7 commits into
flagos-ai:v0.2.0-rc0from
tonyh168:hotfix/v0.2.0
Closed

[hot-fix] 针对2.0.0 修复 flagrelease 发布中遇到的两个bug。#350
tonyh168 wants to merge 7 commits into
flagos-ai:v0.2.0-rc0from
tonyh168:hotfix/v0.2.0

Conversation

@tonyh168

@tonyh168 tonyh168 commented Aug 7, 2026

Copy link
Copy Markdown

Title
hotfix/v0.2.0: fix GroupedTopKRouterFL routing and MLA decode method name

Body(Markdown)

Summary

基于 v0.2.0 tag 的 hotfix,修复两个运行时 bug。共两个 commit,各修一处。

Bug 1 — GroupedTopKRouterFL 路由报错 (4b6c1b2)

vllm_fl/ops/fused_moe/router.pyGroupedTopKRouterFL._compute_routing
调用了 self._valid_grouping(router_logits),但上游 GroupedTopKRouter
并没有 _valid_grouping 方法 —— 它在 _compute_routing 内部用一个局部闭包
valid_grouping() 做校验。因此启用 grouped top-k 路由(DeepSeek-V2/V3、
GLM-MoE、混元等模型)时会抛 AttributeError

修复:移植 v0.3.0-dev 的做法,用内联的 valid_grouping() 闭包替换该调用
(判断逻辑:num_experts > num_expert_group 且能被整除)。仅移植该最小修复,
不含 v0.3.0-dev 中无关的 call_opCachedOp dispatch 重构。

```python
def valid_grouping() -> bool:
num_experts = router_logits.shape[-1]
if num_experts <= self.num_expert_group:
return False
return num_experts % self.num_expert_group == 0

if not valid_grouping():
...
```

Bug 2 — MLA decode 方法签名对不上 (2bac207)

vllm_fl/dispatch/backends/flaggems/impl/mla.pyMLAFLImpl 继承自
MLACommonImpl。vllm 0.20.0 把基类的 decode 覆写方法从 _forward_decode
重命名为 forward_mqa(参数签名完全相同)。FL 侧仍定义 _forward_decode,
导致该实现永远不会被调用,decode 时命中基类抽象的 forward_mqa → 抛
NotImplementedError

修复:将 MLAFLImpl._forward_decode 重命名为 forward_mqa,方法体不变。
(metax 后端用的是自带 vendored 的 MLA,不受影响,未改动。)

Test

  • 两个改动文件均通过 python -m py_compile
  • 确认 router.py 中不再存在 self._valid_grouping 调用
  • 确认 mla.py 中 decode 方法名为 forward_mqa

BrianPei and others added 7 commits May 25, 2026 10:25
# Description

Add CUDA benchmark smoke coverage for vLLM bench in CI. This PR
introduces a reusable benchmark workflow, extends the unified test
runner with a benchmark scope, and adds pytest-based smoke tests for
throughput, latency, and serve.

The new benchmark stage runs after functional tests and validates
benchmark entrypoints, command execution, and result outputs with
lightweight CUDA benchmark cases.

## Type of change

- [x] New feature (non-breaking change which adds functionality)
- [x] Infra/Build change (changes to CI/CD workflows or build scripts)
- [x] Code refactoring
- [ ] Bug fix
- [ ] Documentation change
- [ ] Breaking change

## Changes

- Add reusable CI workflow for benchmark smoke tests
- Add benchmark stage to the platform test pipeline
- Extend run.py with benchmark test discovery and execution
- Add pytest smoke tests for throughput, latency, and serve
- Add shared benchmark helpers for CLI argument generation and command
execution
- Add CUDA A100 benchmark smoke cases with local model/tokenizer paths

## Checklist

- [x] I have read and followed the contributing guidelines
- [x] The functionality is complete
- [x] I have commented my code, particularly in CI workflow setup steps
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added/updated tests that prove my feature works on CUDA
platform
- [x] New and existing unit tests pass locally on CUDA platform

---------

Co-authored-by: HermiaHuan <3081497279@qq.com>
Co-authored-by: XMing <xmhubj@gmail.com>
### PR Category
Tools

### PR Type
Performance

### Description

This benchmark script runs throughput/latency tests against a running
vLLM serve endpoint and exports both raw and averaged metrics to CSV
files.

**How to use**

1. Start the vLLM server

Use a command like:

```bash
vllm serve /models/Qwen3.6-27B --tensor-parallel-size 2 --max-model-len 262144 --no-enable-log-requests --no-enable-prefix-caching
```

2. Run benchmark with default cases

```bash
python benchmarks/benchmark_throughput_serve.py
```

Each test case is:

`(random_input_len, random_output_len, max_concurrency, num_prompts)`

- `random_input_len`: prefill token length per request
- `random_output_len`: generated token length per request
- `max_concurrency`: max concurrent requests
- `num_prompts`: total requests for that benchmark run

Output files

- `raw_runs_<timestamp>.csv`: one row per run
- `summary_<timestamp>.csv`: averaged row per test case (after warmup
skip)

```
# cat summary_20260522_094402.csv 
Prefill,Decode,Conc,Num Prompts,Benchmark Duration (s),Total Input Tokens,Total Output Tokens,Req/s,Output tok/s,Peak Output tok/s,Total tok/s,Mean TTFT (ms),Median TTFT (ms),P99 TTFT (ms),Mean TPOT (ms),Median TPOT (ms),P99 TPOT (ms),Mean ITL (ms),Median ITL (ms),P99 ITL (ms)
1024,1024,64,256,73.59,262144.0,262144.0,3.48,3562.08,4288.0,7124.16,940.97,981.23,2154.05,17.05,17.1,17.9,17.05,16.13,17.89
4096,1024,64,256,107.21,1048576.0,262144.0,2.39,2445.2,3840.0,12225.99,2117.0,1347.29,8819.76,24.08,25.08,25.93,24.08,17.99,269.88
```
### Description
Support for hygon adap, You must set the FlagCX path in bash before you
use vllm-plugin-FL,
such as:  export FLAGCX_PATH=/workspace/FlagCX/
Note: This modification and test depends on FlagGems [PR
#3477](flagos-ai/FlagGems#3477)

### Changes
Add  hygon device in VENDOR_DEVICE_MAP
Add hygon.yaml in vllm_fl/dispatch/config

### Testing
Run Qwen3.5-35B-A3B、Qwen3.6-35B-A3B、Qwen3.6-27B  success。
### PR Category
Core
### PR Type
Bug Fixes

### Description
Fix method miss when setting USE_FLAGGEMS=false.
Block gems op index_put and nonzero to skip error while benchmark e2e
case of 16k_1k.
### PR Category
Vendor

### PR Type
Bug Fixes

### Description
Adapt for hygon

### Related Issues
<!-- Link any related issues: Fixes #issue, Closes #issue, or Related to
#issue -->

### Changes
<!-- List the key changes made in this PR. -->
-

### Testing
<!-- How has this change been tested? Include test commands, hardware
used, etc. -->
-

### Checklist
- [ ] I have run the existing tests and they pass
- [ ] I have added tests for my changes (if applicable)
- [ ] I have updated the documentation (if applicable)

Signed-off-by: ftgreat <ldwang@baai.ac.cn>
GroupedTopKRouterFL._compute_routing called self._valid_grouping(), which
does not exist on upstream GroupedTopKRouter (it uses an inner valid_grouping()
closure), raising AttributeError during grouped top-k routing. Port the inline
closure from v0.3.0-dev.
vllm 0.20.0 renamed the MLACommonImpl decode override from _forward_decode
to forward_mqa (same signature). MLAFLImpl still defined _forward_decode, so
its implementation was never dispatched and the abstract forward_mqa raised
NotImplementedError at decode. Rename to match the base class.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 5 committers have signed the CLA.

✅ cyber-pioneer
✅ creaspoggg
✅ BrianPei
✅ ftgreat
❌ loopyt


loopyt seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants