[FR] Bug: Abnormal FlagGems ON/OFF accuracy gap in Plugin mode (Phi-3-mini-128k: ON=30%, OFF=42%)

[buzhengjing]

Summary

When running LLM-Research/Phi-3-mini-128k-instruct in Plugin mode (V3), abnormal behavior observed:

• V2 (no Plugin, FlagGems ON): 36.0% ✓
• V3 (Plugin + FlagGems ON): 30.0% — 9.1% relative degradation > 5%
• V3 (Plugin + FlagGems OFF): 42.0% — all-off is actually better!
• V3 (Plugin, attention_backend disabled): 26.0% — worse

Key finding: In Plugin mode, FlagGems ON (30%) vs OFF (42%) shows a 12 percentage point gap, indicating FlagGems operators produce negative interactions when dispatched through the Plugin path.

Environment

Item	Value
GPU	NVIDIA H20-3e ×8
PyTorch	2.11.0+cu130
vllm-plugin-FL	0.0.0
FlagGems	5.0.2
vLLM	0.20.2
Model	LLM-Research/Phi-3-mini-128k-instruct (3.8B, 128k ctx)
Benchmark	GPQA Diamond (50 questions, greedy)

Steps to Reproduce

1. pip install vllm-plugin-FL
2. VLLM_PLUGINS=fl USE_FLAGGEMS=1 vllm serve Phi-3-mini-128k-instruct --max-model-len 32768
3. Run GPQA Diamond → score 30.0%
4. Change to USE_FLAGGEMS=0 → score 42.0%

Accuracy Comparison Matrix

Mode	Configuration	Score	vs NV Baseline (33%)
V2 (no Plugin)	FlagGems ON	36.0%	+9.1% ✓
V3 Plugin	FlagGems ON (full)	30.0%	-9.1% ✗
V3 Plugin	FlagGems OFF	42.0%	+27.3% ✓
V3 Plugin	attention_backend disabled	26.0%	-21.2% ✗

Analysis

1. V2 (no Plugin) + FlagGems = 36% vs V3 (Plugin) + FlagGems = 30% → Plugin introduces 6pp degradation
2. V3 (Plugin) + FlagGems OFF = 42% → Plugin itself does NOT harm accuracy, may even help
3. Core issue: FlagGems operators produce negative interaction when dispatched through Plugin OpManager

Suggested Investigation

1. Plugin OpManager dispatch vs FlagGems ops registration order/priority conflict
2. Phi-3-mini-128k LongRoPE extension compatibility with Plugin+FlagGems combination
3. 128k context model attention path special handling in Plugin mode
4. Compare V2 FlagGems injection method (code injection) vs V3 Plugin dispatch path differences