Skip to content

refactor(musa): remove direct flaggems calls from vendor backend - #376

Merged
cyber-pioneer merged 1 commit into
v0.3.0-devfrom
fix/musa-remove-flaggems-from-vendor
Aug 14, 2026
Merged

refactor(musa): remove direct flaggems calls from vendor backend#376
cyber-pioneer merged 1 commit into
v0.3.0-devfrom
fix/musa-remove-flaggems-from-vendor

Conversation

@physics31415926

@physics31415926 physics31415926 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Refactor MUSA backend to remove direct FlagGems calls for topk_softmax. This method is now dispatched through the standard op manager system.

Changes

  • Removed topk_softmax method from MusaBackend class
  • topk_softmax now routes through op manager dispatch system, using FlagGems implementation when available

Testing

Tested on MUSA hardware (moer_15 machine) with MoE model inference:

Environment:

  • Hardware: MTT S5000 (8x 80GB GPUs)
  • Driver: 3.3.5-server
  • Container: vllm_graph_torch29_new
  • Model: Qwen3.6-35B-A3B (67GB MoE model)

Test Configuration:

  • Tensor Parallel: 4 (GPU 2-5)
  • Max model length: 2048
  • Mode: Eager execution
  • GPU memory utilization: 0.9

Result: ✅ PASSED

  • Model loaded successfully (67GB in 15.5s)
  • topk_softmax correctly dispatched to FlagGems: Patched vllm_topk_softmax for MUSA (FlagGems) in: fused_topk_router, fused_topk_bias_router
  • Inference successful with correct outputs:
    • "Hello, my name is" → "John. I am a 25-year-old male, and I have been experiencing a persistent cough"
    • "The capital of France is" → "Paris, a city renowned for its iconic landmarks such as the Eiffel Tower, the Louvre Museum"

Performance:

  • Throughput: ~3.6 toks/s output, ~0.9 toks/s input
  • KV cache: 2.2M tokens allocated
  • Max concurrency: 1086x for 2048-token requests

@github-actions github-actions Bot added the core label Aug 13, 2026
@cyber-pioneer
cyber-pioneer merged commit c776ba1 into v0.3.0-dev Aug 14, 2026
2 checks passed
@physics31415926
physics31415926 deleted the fix/musa-remove-flaggems-from-vendor branch August 18, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants