Skip to content

feat(xpu): Intel XPU support for SGLang-Omni#994

Draft
siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:feat/intel-xpu-support
Draft

feat(xpu): Intel XPU support for SGLang-Omni#994
siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:feat/intel-xpu-support

Conversation

@siju-samuel

@siju-samuel siju-samuel commented Jul 8, 2026

Copy link
Copy Markdown

Motivation

Add Intel GPUs (XPU) support alongside the NVIDIA CUDA backend.
Qwen3-ASR, Qwen3-TTS, and Qwen3-Omni all serve end-to-end on Arc Pro B60
ASR and TTS single-card,
Qwen3-Omni 30B-A3B thinker across 8 cards with tensor parallelism.

Modifications

  • Device abstraction (sglang_omni/utils/device.py): resolve the active accelerator (cuda/xpu/cpu), wrap the torch.cuda.* surface, and add a caps() capability registry so features gate on capability, not device identity. On CUDA it resolves to the exact prior calls (no regression).
  • Serve / pipeline / runner: --device {cuda,xpu} flag; device-agnostic relay device string and set_device; multi-XPU TP device mapping (all cards visible + real gpu_id on XPU for XCCL); build ForwardBatch directly on sglang 0.5.13.
  • SGLang 0.5.12 -> 0.5.13 compatibility: post-hoc memory-pool / attention / lora / cuda-graph init and renamed-attr backfill; 0.5.13 scheduler-state init and output-streamer routing; tolerant ServerArgs kwargs reconciler.
  • XPU backend policy (central ModelWorker): SGLang selects the graph runner per device (xpu -> XPUGraphRunner, a torch.compile wrapper that stays eager and is None-safe on the forward path), so the graph path is left to SGLang rather than force-disabled. The CUDA-only kernel fallbacks — triton MoE runner (no FP8 CUTLASS) and fused_qk_norm_rope-off — are gated on their own caps (sgl_kernel/fp8), keeping them independent of the graph decision. No-op on CUDA.
  • Seeded sampler (sglang_omni/sampling/xpu_sampler.py): pure-torch, bit-identical murmur_hash32 / multinomial_with_seed for XPU, rebinding SGLang's Triton path only on XPU (CUDA unchanged). Wired into the central ModelWorker XPU policy.
  • Per-model device fixes for qwen3_asr / qwen3_omni / qwen3_tts: resolve device strings via the device layer, gate CUDA-only fast paths, run the Mimi codec on CPU on XPU (one-shot, off the hot path; the full KV pool leaves no device headroom for it), and other XPU-safe adjustments.
  • Qwen3-Omni multi-XPU tensor parallelism (sglang_omni/utils/xpu_sglang_compat.py): correct SGLang's XPU free-memory accounting to real free memory (it over-reported total-minus-allocator, over-sizing the KV pool); raise the post-load TP barrier timeout for slow multi-rank shard loads; build the thinker request's origin_input_ids as array("q", ...) and wrap the custom omni forward in a 0.5.13 forward_context; tolerate the is_chunked -> inflight_middle_chunks rename.
  • Packaging: pyproject_xpu.toml (+xpu torch pins, lean Qwen3 core + extras), scripts/xpu/install_xpu.sh, docker/xpu.Dockerfile.
  • Docs / tests: README Hardware Support + Intel XPU install guide; device-layer, seeded-sampler, and XPU-feasibility unit tests under tests/unit_test/xpu/.

Related Issues

Accuracy Test

Benchmark & Profiling

Checklist

  • Format your code according with pre-commit.
  • Add unit tests.
  • Update documentation / docstrings / example tutorials as needed.
  • Provide throughput / latency benchmark results and accuracy evaluation results as needed.
  • For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.

CI

CI runs on self-hosted GPU runners and requires a maintainer to add the
run-ci label. Once labeled, every subsequent push re-triggers CI as
long as the label remains. Use /tag-and-rerun-ci higgs or
/tag-and-rerun-ci moss to select a TTS CI model. Draft PRs are skipped even
if labeled.

cc: @rbabukv

Add Intel Arc GPU (XPU) support alongside the NVIDIA CUDA backend, via PyTorch
XPU. Qwen3-ASR, Qwen3-TTS, and Qwen3-Omni all serve end-to-end on Arc Pro B60 —
ASR and TTS single-card, the Qwen3-Omni 30B-A3B thinker across 8 cards with
tensor parallelism.

- Device abstraction (sglang_omni/utils/device.py): resolve the active
  accelerator (cuda/xpu/cpu), wrap the torch.cuda.* surface, and add a caps()
  capability registry so features gate on capability, not device identity. On
  CUDA it resolves to the exact prior calls (no regression).
- Serve / pipeline / runner: --device {cuda,xpu} flag; device-agnostic relay
  device string and set_device; multi-XPU TP device mapping (all cards visible +
  real gpu_id on XPU for XCCL); build ForwardBatch directly on sglang 0.5.13.
- SGLang 0.5.12 -> 0.5.13 compatibility: post-hoc memory-pool / attention /
  lora / cuda-graph init and renamed-attr backfill; 0.5.13 scheduler-state init
  and output-streamer routing; tolerant ServerArgs kwargs reconciler.
- XPU backend policy (central ModelWorker): SGLang selects the graph runner per
  device (xpu -> XPUGraphRunner, a torch.compile wrapper that stays eager and is
  None-safe on the forward path), so the graph path is left to SGLang rather than
  force-disabled. The CUDA-only kernel fallbacks — triton MoE runner (no FP8
  CUTLASS) and fused_qk_norm_rope-off — are gated on their own caps
  (sgl_kernel/fp8), keeping them independent of the graph decision. No-op on CUDA.
- Seeded sampler (sglang_omni/sampling/xpu_sampler.py): pure-torch, bit-identical
  murmur_hash32 / multinomial_with_seed for XPU, rebinding SGLang's Triton path
  only on XPU (CUDA unchanged). Wired into the central ModelWorker XPU policy.
- Per-model device fixes for qwen3_asr / qwen3_omni / qwen3_tts: resolve device
  strings via the device layer, gate CUDA-only fast paths, run the Mimi codec on
  CPU on XPU (one-shot, off the hot path; the full KV pool leaves no device
  headroom for it), and other XPU-safe adjustments.
- Qwen3-Omni multi-XPU tensor parallelism (sglang_omni/utils/xpu_sglang_compat.py):
  correct SGLang's XPU free-memory accounting to real free memory (it over-reported
  total-minus-allocator, over-sizing the KV pool); raise the post-load TP barrier
  timeout for slow multi-rank shard loads; build the thinker request's
  origin_input_ids as array("q", ...) and wrap the custom omni forward in a 0.5.13
  forward_context; tolerate the is_chunked -> inflight_middle_chunks rename.
- Packaging: pyproject_xpu.toml (+xpu torch pins, lean Qwen3 core + extras),
  scripts/xpu/install_xpu.sh, docker/xpu.Dockerfile.
- Docs / tests: README Hardware Support + Intel XPU install guide; device-layer,
  seeded-sampler, and XPU-feasibility unit tests under tests/unit_test/xpu/.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant