feat(xpu): Intel XPU support for SGLang-Omni#994
Draft
siju-samuel wants to merge 1 commit into
Draft
Conversation
Add Intel Arc GPU (XPU) support alongside the NVIDIA CUDA backend, via PyTorch
XPU. Qwen3-ASR, Qwen3-TTS, and Qwen3-Omni all serve end-to-end on Arc Pro B60 —
ASR and TTS single-card, the Qwen3-Omni 30B-A3B thinker across 8 cards with
tensor parallelism.
- Device abstraction (sglang_omni/utils/device.py): resolve the active
accelerator (cuda/xpu/cpu), wrap the torch.cuda.* surface, and add a caps()
capability registry so features gate on capability, not device identity. On
CUDA it resolves to the exact prior calls (no regression).
- Serve / pipeline / runner: --device {cuda,xpu} flag; device-agnostic relay
device string and set_device; multi-XPU TP device mapping (all cards visible +
real gpu_id on XPU for XCCL); build ForwardBatch directly on sglang 0.5.13.
- SGLang 0.5.12 -> 0.5.13 compatibility: post-hoc memory-pool / attention /
lora / cuda-graph init and renamed-attr backfill; 0.5.13 scheduler-state init
and output-streamer routing; tolerant ServerArgs kwargs reconciler.
- XPU backend policy (central ModelWorker): SGLang selects the graph runner per
device (xpu -> XPUGraphRunner, a torch.compile wrapper that stays eager and is
None-safe on the forward path), so the graph path is left to SGLang rather than
force-disabled. The CUDA-only kernel fallbacks — triton MoE runner (no FP8
CUTLASS) and fused_qk_norm_rope-off — are gated on their own caps
(sgl_kernel/fp8), keeping them independent of the graph decision. No-op on CUDA.
- Seeded sampler (sglang_omni/sampling/xpu_sampler.py): pure-torch, bit-identical
murmur_hash32 / multinomial_with_seed for XPU, rebinding SGLang's Triton path
only on XPU (CUDA unchanged). Wired into the central ModelWorker XPU policy.
- Per-model device fixes for qwen3_asr / qwen3_omni / qwen3_tts: resolve device
strings via the device layer, gate CUDA-only fast paths, run the Mimi codec on
CPU on XPU (one-shot, off the hot path; the full KV pool leaves no device
headroom for it), and other XPU-safe adjustments.
- Qwen3-Omni multi-XPU tensor parallelism (sglang_omni/utils/xpu_sglang_compat.py):
correct SGLang's XPU free-memory accounting to real free memory (it over-reported
total-minus-allocator, over-sizing the KV pool); raise the post-load TP barrier
timeout for slow multi-rank shard loads; build the thinker request's
origin_input_ids as array("q", ...) and wrap the custom omni forward in a 0.5.13
forward_context; tolerate the is_chunked -> inflight_middle_chunks rename.
- Packaging: pyproject_xpu.toml (+xpu torch pins, lean Qwen3 core + extras),
scripts/xpu/install_xpu.sh, docker/xpu.Dockerfile.
- Docs / tests: README Hardware Support + Intel XPU install guide; device-layer,
seeded-sampler, and XPU-feasibility unit tests under tests/unit_test/xpu/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Add Intel GPUs (XPU) support alongside the NVIDIA CUDA backend.
Qwen3-ASR, Qwen3-TTS, and Qwen3-Omni all serve end-to-end on Arc Pro B60
ASR and TTS single-card,
Qwen3-Omni 30B-A3B thinker across 8 cards with tensor parallelism.
Modifications
Related Issues
Accuracy Test
Benchmark & Profiling
Checklist
CI
CI runs on self-hosted GPU runners and requires a maintainer to add the
run-cilabel. Once labeled, every subsequent push re-triggers CI aslong as the label remains. Use
/tag-and-rerun-ci higgsor/tag-and-rerun-ci mossto select a TTS CI model. Draft PRs are skipped evenif labeled.
cc: @rbabukv