[Doc][serve] Recipe for DeepSeek-V4 on NPU via Ray Serve + vLLM

[Artimislyy]

Description

Add NPU (Huawei Ascend) accelerator backend support for Ray Serve LLM and a step-by-step deployment guide for serving DeepSeek-V4-Flash-w8a8-mtp on Atlas 800 A2/A3 servers using vLLM-Ascend.

Code:

accelerators.py: NPUConfig, NPUAccelerator, NPU_ACCELERATOR_VALUES, bundle inference for NPU
llm_config.py: NPU resolution and hardware mismatch validation
vllm_models.py: NPUConfig → NPUAccelerator instantiation
test_models.py: NPU unit tests (config construction, inference, mismatch validation)

Docs:

npu-ascend.md: DeepSeek-V4 deployment guide on Ascend NPU
examples.md: NPU entry in "By capability"

Related issues

#62983

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive guide for deploying DeepSeek-V4 on Huawei Ascend NPUs using Ray Serve and vLLM. The review feedback identifies a potential API misuse in the build_openai_app function, suggests handling kernel parameter tuning on the host machine to avoid permission issues within Docker, and recommends moving environment variables into the runtime_env configuration to ensure consistent application across distributed Ray Serve replicas.

[gemini-code-assist]

The build_openai_app function typically expects a list of LLMConfig objects or a single LLMConfig object directly. Passing a dictionary with an llm_configs key is likely incorrect for the public Ray Serve LLM API and may cause initialization errors or result in an incorrect model name being exposed in the OpenAI-compatible API.

Suggested change

	app = build_openai_app({"llm_configs": [llm_config]})
	app = build_openai_app([llm_config])

[Artimislyy]

The referenced is Deploy a medium-sized LLM

[gemini-code-assist]

These sysctl commands for kernel parameter tuning require root privileges and will likely fail when executed inside a standard Docker container. It is recommended to apply these settings on the host machine before starting the container. If they must be run inside the container, it needs to be started with the --privileged flag or specific capabilities (e.g., --cap-add=SYS_ADMIN), though host-side configuration is preferred for security.

[Artimislyy]

Updated.

[gemini-code-assist]

To ensure that all NPU-specific performance and memory management settings are consistently applied across all Ray Serve replicas (especially in multi-node or distributed environments where the shell environment might not be inherited), it is recommended to include the environment variables defined in Step 3 within the runtime_env of the LLMConfig.

Suggested change

	runtime_env={
	"env_vars": {
	"VLLM_USE_V1": "1",
	}
	},
	runtime_env={
	"env_vars": {
	"VLLM_USE_V1": "1",
	"PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True",
	"ACL_OP_INIT_MODE": "1",
	"VLLM_ASCEND_ENABLE_FLASHCOMM1": "1",
	"USE_MULTI_GROUPS_KV_CACHE": "1",
	"TASK_QUEUE_ENABLE": "1",
	"HCCL_OP_EXPANSION_MODE": "AIV",
	"HCCL_BUFFSIZE": "512",
	"USE_MULTI_BLOCK_POOL": "1",
	}
	},

[Artimislyy]

Since A2 and A3 need different environment variables and kernel parameters, I recommend configuring them inside the Docker container before starting Ray.

[KaisennHu]

We should explicitly document the versions of critical software dependencies for clarity.

[Artimislyy]

This has been added to Step 2: Start the Docker Container.

[tianyi-ge]

is it necessary to modify here? resources like tpu/npu can be automatically resolved by **ray_actor_options.get("resources", {}),

[Artimislyy]

The original code unconditionally declares "GPU": 0 in replica_actor_resources, even on NPU deployments. This produces:

{"CPU": 1, "GPU": 0, "NPU": 8}

[tianyi-ge]

PTAL at the microcheck ci. be sure to do pre-commit check

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[jeffreywang88]

@Artimislyy @tianyi-ge do you have any updates for this PR? would love to get NPU support for ray serve LLM!

[Artimislyy]

@Artimislyy @tianyi-ge do you have any updates for this PR? would love to get NPU support for ray serve LLM!

Today, verify again on the NPU whether there are any issues.

[kouroshHakha]

Can we put this in the ray serve llm examples section? https://docs.ray.io/en/latest/serve/llm/examples.html

[Artimislyy]

Sure, but the latest Ray version has some compatibility issues with vLLM and vllm-ascend right now. I'll provide more details.

[Artimislyy]

Can we put this in the ray serve llm examples section? https://docs.ray.io/en/latest/serve/llm/examples.html

Ray Serve LLM currently lacks NPU accelerator type support. Once this PR is merged, LLM inference services can be directly deployed on Ascend NPU clusters. There are currently two options available.
Option 1: Deploy Now (requires manual patching)

Environment versions: vllm 0.22.0 / vllm-ascend 0.21.0 / CANN 9.0.0 / torch 2.10.0 / torch_npu 2.10.0 / torchvision 2.25.0

Known limitations:

vllm 0.22.0 has NPU compatibility bugs (fixed in vllm 0.24.0), requiring manual source code patches to function properly
triton-ascend cannot enable graph-mode acceleration; standard inference works normally. An issue will be filed with the triton-ascend community
Option 2: Wait for upstream release (zero-patch deployment)

Wait for vllm-ascend 0.24.0 to be officially released. It ships with vllm 0.24.0 which has already fixed the aforementioned NPU bugs. At that point, this PR's code combined with the vllm-ascend 0.24.0 image will enable direct deployment without any source code modifications.
@jeffreywang88

[jeffreywang88]

is vllm-ascend 0.24.0 released?

[jeffreywang88]

we should have a solution that doesn't require patching in the doc

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 8a97a3d. Configure here.}

[cursor]

NPU bundles include GPU zero

Medium Severity

New NPUAccelerator default bundles request NPU devices, but LLMServer.get_deployment_options still merges every replica bundle with "GPU": 0, so NPU deployments (including the tutorial’s default accelerator_config) get placement groups like {CPU, GPU: 0, NPU}.

 

^{Reviewed by Cursor Bugbot for commit 8a97a3d. Configure here.}

[Artimislyy]

The merge in _merge_replica_actor_and_child_actor_bundles produces bundles like {"NPU": 1, "CPU": 5, "GPU": 0}, but Ray's scheduling comparison in ResourceSet::operator<= (source) checks this_value > other_value for each resource key. Since this_value for GPU is 0, the condition 0 > anything is always false — it never requires the node to actually have GPUs. The NPU bundle can be scheduled to a pure NPU node without issue.

[jeffreywang88]

don't we need ray[llm]? can you paste a screenshot for the server and the client sending request and receiving responses?

[Artimislyy]

don't we need ray[llm]? can you paste a screenshot for the server and the client sending request and receiving responses?

I didn't install Ray Serve or Ray LLM. I downloaded the latest Ray from the daily builds and added NPU-related modifications. It worked because the vllm-ascend image already ships vllm and runtime dependencies. However, pip install "ray[llm]" is the correct and safer approach. I'm verifying this on our internal network, so I can't share the images externally.

[jeffreywang88]

could we provide a dockerfile in the example (maybe not directly in the markdown, but somewhere nearby) so that users only need to write ray serve llm application code to run the example?

[Artimislyy]

could we provide a dockerfile in the example (maybe not directly in the markdown, but somewhere nearby) so that users only need to write ray serve llm application code to run the example?

Great news: vllm-ascend v0.22.1 is released, and deploying DeepSeek-V4 no longer requires manually patching vLLM. Regarding your suggestion of baking environment variables into a Dockerfile — tuning parameters vary significantly across different models, and hardcoding them would make the image only suitable for a single model.

[jeffreywang88]

good to know! does this mean that a Ray Serve LLM deployment will succeed if I start a container using quay.io/ascend/vllm-ascend:v0.22.1rc1 and then run pip install ray[llm]? can we provide a link to vllm-ascend documentation for where to find the latest image?

are all of these environment variables and docker run arguments necessary? if they vary across models and deployment strategy (e.g. kuberay or VM), we should remove them.

[Artimislyy]

does this mean that a Ray Serve LLM deployment will succeed if I start a container using quay.io/ascend/vllm-ascend:v0.22.1rc1 and then run pip install ray[llm]?

Yes, but first you need to merge this PR to enable Ray Serve to support NPU as an LLM backend.

can we provide a link to vllm-ascend documentation for where to find the latest image?

Done — added to the npu-ascend.md doc, in Step 2.

all of these environment variables and docker run arguments necessary? if they vary across models and deployment strategy (e.g. kuberay or VM), we should remove them.

These environment variables and docker run arguments vary by model and hardware, but the npu-ascend.md doc notes that users should consult the vLLM-Ascend documentation and adjust them based on the model and deployment scenario. The environment variables in the npu-ascend.md doc are provided as an example for DeepSeek-V4-Flash on a single-node A2 deployment.

[jeffreywang88]

can this be fixed upstream?

[jeffreywang88]

could you please include screenshots / logs for an actual ray serve LLM NPU deployment similar to #65026 (comment)?

[jeffreywang88]

does a later ray version support NPU?

[Artimislyy]

does a later ray version support NPU?

Ray Serve does not natively support NPU as an LLM backend; you need to merge the code from this PR.

[jeffreywang88]

Gotcha. Can you adjust the descriptions to reflect the post-PR state -- the state after these changes have been merged?

[Artimislyy]

Done.

[jeffreywang88]

good to know! does this mean that a Ray Serve LLM deployment will succeed if I start a container using quay.io/ascend/vllm-ascend:v0.22.1rc1 and then run pip install ray[llm]? can we provide a link to vllm-ascend documentation for where to find the latest image?

are all of these environment variables and docker run arguments necessary? if they vary across models and deployment strategy (e.g. kuberay or VM), we should remove them.

[jeffreywang88]

can you unify test_npu_accelerator_type_hardware_mismatch_with_gpu_config and test_gpu_accelerator_type_hardware_mismatch_with_npu_config with pytest.mark.parametrize?

[Artimislyy]

can you unify test_npu_accelerator_type_hardware_mismatch_with_gpu_config and test_gpu_accelerator_type_hardware_mismatch_with_npu_config with pytest.mark.parametrize?

Done.

[Artimislyy]

could you please include screenshots / logs for an actual ray serve LLM NPU deployment similar to #65026 (comment)?

(base) [root@90-90-93-228 ~]# curl http://localhost:8900/v1/chat/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "dsv4",
>         "messages": [
>             {
>                 "role": "user",
>                 "content": "Who are you?"
>             }
>         ],
>         "max_tokens": 256,
>         "temperature": 0
>     }'
{"id":"chatcmpl-bae55e8f007cb28e","object":"chat.completion","created":1784081972,"model":"dsv4","choices":[{"index":0,"message":{"role":"assistant","content":"Hi there! I'm DeepSeek, an AI assistant created by the company DeepSeek (深度求索). I'm here to help you with a wide range of tasks—whether it's answering questions, brainstorming ideas, writing, coding, or just having a friendly chat.\\n\\nI'm a text-based model with a knowledge cutoff in May 2025, and I can handle up to 1 million tokens of context, which means I can process very long documents or conversations in one go. I'm also capable of reading files (like PDFs, Word docs, Excel sheets, images with text, and more) and can search the web if you enable that feature.\\n\\nThe best part? I'm completely free to use! So feel free to ask me anything—I'm here to help. 😊\\n\\nWhat can I do for you today?"},"refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"routed_experts":null}],"service_tier":null,"system_fingerprint":"vllm-0.22.1-tp8-ep-70a4bea1","usage":{"prompt_tokens":8,"total_tokens":181,"completion_tokens":173,"prompt_tokens_details":null,"completion_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"prompt_text":null,"kv_transfer_params":null}
(base) [root@90-90-93-228 ~]#

[dstrodtman]

Docs-team style pass from Douglas Strodtman (@dstrodtman, Anyscale docs team). Claude Code assisted with this review; I read every comment below and stand behind it.

Scope, and what I'm staying out of. Style, grammar, and Ray documentation conventions only, against the Ray documentation style guide. @jeffreywang88 and @kouroshHakha have several open threads on this PR about substance: whether the recipe should require patching, whether a Dockerfile belongs alongside it, whether all the environment variables are needed, and whether the prose describes the post-merge state. I'm not weighing in on any of those and nothing below re-raises them. They're the maintainers' calls and they gate this PR; my comments don't.

Thanks for the page. A working Ascend NPU recipe is genuinely useful, and the step structure is the right shape for it.

Two findings I'd single out as worth more than ordinary style attention:

1. Two docs.ray.io/en/latest/... URLs point at pages in this same repo. This is more than a style preference. /en/latest/ pins the reader to the last release, so a reader on master docs silently leaves the version they're browsing, and the link rots quietly instead of failing the build. Sphinx cross-references resolve at build time and survive page moves. I've given you the exact targets, both taken from links that already exist in this PR or its directory rather than guessed.

2. The curl example uses a model name the config never defines. model_loading_config sets model_id="deepseek-v4-flash", and the request in Step 6 asks for "model": "dsv4". As written, someone following the guide end to end gets an error on the last step. I've flagged it as a question rather than suggesting a fix, because I don't know which of the two you intended.

The rest is mechanical: missing html_meta front matter, five title-case headings (the guide asks for sentence case, and nothing in CI catches this), refer to for See, e.g. and vs. and etc., one It is recommended to, and a > **Note:** blockquote where MyST has a real admonition.

One thing I noticed but am not filing: the code fences use sh, where quick-start.md and the rest of this directory use bash. Worth matching, not worth a comment of its own.

Happy to stamp the docs side once the maintainers are satisfied with the substance. To be explicit: I'm not blocking anything, and if a maintainer wants this in with the prose as-is, that's entirely reasonable.

[dstrodtman]

The style guide asks you to give every page a description in its MyST html_meta front matter, which search engines and the page's social preview use. This page has no front matter at all.

Worth knowing that this directory is inconsistent: index.md has one, and quick-start.md, examples.md, benchmarks.md, and troubleshooting.md don't. So you'd be ahead of your neighbors rather than matching them. I'm asking anyway, because a new page is the cheap moment to add it and this one has an unusually specific audience worth reaching: someone searching for Ascend NPU or vLLM-Ascend serving won't find much else.

I matched the phrasing style of index.md, the one page here that has a description.

Suggested change

	# Deploy an LLM on Ascend NPU
	---
	myst:
	html_meta:
	description: "Deploy DeepSeek-V4-Flash-w8a8-mtp on Huawei Ascend NPUs with Ray Serve LLM and vLLM-Ascend: Docker setup, NPU environment variables, Ray cluster startup, and an OpenAI-compatible endpoint."
	---
	# Deploy an LLM on Ascend NPU

Adjust the wording freely. The only thing I'd keep is naming both Ascend NPU and vLLM-Ascend, since those are the terms a reader would search for.

[dstrodtman]

Three things here, and the link is the one that matters.

The docs.ray.io/en/latest/ link points into this repo. It resolves to "Deploy a medium-sized LLM," which examples.md in this same PR already links as {doc} with an absolute path. Using the cross-reference means the link tracks page moves and stays on the reader's version instead of jumping them to the last release.

Cut the benefits clause. enabling scalable, efficient, and OpenAI-compatible LLM serving on Ascend NPU hardware is the kind of benefits list the guide tells you to cut. The reader is already here; they don't need to be sold.

Split the sentence. It's about 90 words carrying two links and a parenthetical, and the guide asks you to open a page with one or two sentences that say what it covers.

Suggested change

This guide provides a step-by-step recipe for deploying DeepSeek-V4-Flash-w8a8-mtp (w8a8 quantized with multi-token prediction) on Huawei Ascend NPUs using [Ray Serve LLM](https://docs.ray.io/en/latest/serve/tutorials/deployment-serve-llm/medium-size-llm/README.html) and [vLLM-Ascend](https://github.com/vllm-project/vllm-ascend), enabling scalable, efficient, and OpenAI-compatible LLM serving on Ascend NPU hardware. If you want to deploy other large language models, you can combine the approach in this guide with the deployment solutions for other models provided in the [vLLM-Ascend documentation](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/index.html).

This guide is a step-by-step recipe for deploying DeepSeek-V4-Flash-w8a8-mtp on Huawei Ascend NPUs with {doc}`Ray Serve LLM </_collections/serve/tutorials/deployment-serve-llm/medium-size-llm/README>` and [vLLM-Ascend](https://github.com/vllm-project/vllm-ascend). The model is w8a8 quantized with multi-token prediction. To deploy a different model, combine the approach here with the per-model deployment guidance in the [vLLM-Ascend documentation](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/index.html).

I took the {doc} target verbatim from your own addition to examples.md, so it's the same path that file already uses. If you meant to link the Ray Serve LLM landing page rather than the medium-size-LLM tutorial, that's [Ray Serve LLM](index.md) instead.

[dstrodtman]

The style guide asks for sentence case in headings: first word and proper nouns only. Five of the six Step N: headings need it (Step 5 is already correct), and I've left a one-line suggestion on each. Nothing in Ray's CI catches heading case, so it's a human check every time.

Suggested change

	## Step 1: Download Model Weights
	## Step 1: Download model weights

[dstrodtman]

This reads as a fragment, with the colon doing the work a verb should do. Two smaller things while you're here:

• The guide asks you to spell out zero through nine in prose, so 1 Atlas 800 A3 becomes one Atlas 800 A3. The specs inside the parentheses stay as numerals.
• (Quantized version) duplicates the w8a8 already in the filename.

The line is also missing its closing period.

I've left the 128G × 8 and 64G × 8 sizing exactly as you wrote it.

Suggested change

	`DeepSeek-V4-Flash-w8a8-mtp` (Quantized version): requires 1 Atlas 800 A3 (128G × 8) node or 1 Atlas 800 A2 (64G × 8) node. [Download model weights](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp)
	`DeepSeek-V4-Flash-w8a8-mtp` requires one Atlas 800 A3 node (128G × 8) or one Atlas 800 A2 node (64G × 8). [Download the model weights](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp).

[dstrodtman]

It is recommended to is passive with no actor, where the guide asks you to turn a recommendation into direct advice and to use the imperative for instructions. The line is also missing its closing period.

Suggested change

	It is recommended to download the model weights to a shared directory accessible by multiple nodes, such as `/root/.cache/`
	Download the model weights to a shared directory that all nodes can reach, such as `/root/.cache/`.

[dstrodtman]

On the content of this note, I'm deferring to @jeffreywang88's open thread about describing the post-merge state, and to his related point about not requiring a patch in the doc. Whatever this note ends up saying is between you and him, and my suggestion below keeps your current wording untouched.

What I'm asking for is only the container. The guide asks you to use MyST admonitions rather than a bold inline label, and a blockquote with **Note:** renders as an indented paragraph rather than a callout, so it's easy to skim past. This is a prerequisite that will break the whole recipe if missed, so important fits it better than note.

Suggested change

	> **Note:** The image includes Ray version 2.48.0, which does not support NPU. You need to install a version of Ray that includes NPU support (NPUConfig, NPUAccelerator).
	:::{important}
	The image includes Ray version 2.48.0, which does not support NPU. You need to install a version of Ray that includes NPU support (NPUConfig, NPUAccelerator).
	:::

If that text changes on Jeffrey's thread, keep the :::{important} wrapper and swap the sentence inside it.

[dstrodtman]

Two word-choice fixes and one link:

• e.g., becomes "for example" in prose.
• refer to becomes "See".
• The docs.ray.io/en/latest/serve/llm/index.html URL points at index.md, the page sitting next to this one. A plain Markdown link between .md pages is build-verified and follows the reader's version.

Suggested change

	Create a Python script (e.g., `serve_npu.py`) with the following content. For more details on how to use Ray Serve LLM to deploy LLMs, refer to the [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/index.html).
	Create a Python script, for example `serve_npu.py`, with the following content. For more on deploying LLMs with Ray Serve LLM, see the [Ray Serve LLM documentation](index.md).

[dstrodtman]

Sentence case, as in the Step 1 heading comment.

Suggested change

	## Step 6: Send Requests
	## Step 6: Send requests

[dstrodtman]

cURL is the project's own branding for the tool, but in prose the command is curl. Google's style guide, which Ray falls back to, uses the lowercase command form.

Suggested change

	You can query the deployed model with cURL:
	Query the deployed model with `curl`:

[dstrodtman]

A question rather than a suggestion, because I don't know which value you intended. This request asks for "model": "dsv4", but model_loading_config in Step 5 sets model_id="deepseek-v4-flash". Ray Serve LLM routes on the model_id, so a reader who follows every step in order should get an error on this last one.

Two ways to reconcile it, and you'd know which is right:

• The model_id is what you meant, and this should be "model": "deepseek-v4-flash".
• dsv4 is what you actually ran with, and Step 5's model_id should be dsv4.

I'd lean toward the first, since deepseek-v4-flash is the more self-describing of the two and matches the weights directory, but that's a preference and the correctness call is yours. Worth fixing either way: the last step of a recipe is the one readers file issues about.

[Artimislyy]

@dstrodtman Thanks for the suggestion — done.