[Bug]: NVIDIA DGX Spark vllms and gemma 4 26B won't work

### Your current environment

<details>
<summary>The output of <code>python collect_env.py</code></summary>

```text
(vllm) raymondlo84@spark-ray:~/spark_serving/vllm$ python collect_env.py
/home/raymondlo84/spark_serving/vllm/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from .version import __version__, __version_tuple__  # isort:skip
Traceback (most recent call last):
  File "/home/raymondlo84/spark_serving/vllm/collect_env.py", line 20, in <module>
    from vllm.envs import environment_variables
  File "/home/raymondlo84/spark_serving/vllm/vllm/__init__.py", line 14, in <module>
    import vllm.env_override  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raymondlo84/spark_serving/vllm/vllm/env_override.py", line 90, in <module>
    from vllm.utils.torch_utils import is_torch_equal, is_torch_equal_or_newer
  File "/home/raymondlo84/spark_serving/vllm/vllm/utils/torch_utils.py", line 74, in <module>
    PIN_MEMORY = is_pin_memory_available()
  File "/home/raymondlo84/spark_serving/vllm/vllm/utils/platform_utils.py", line 45, in is_pin_memory_available
    from vllm.platforms import current_platform
  File "/home/raymondlo84/spark_serving/vllm/vllm/platforms/__init__.py", line 294, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raymondlo84/spark_serving/vllm/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname
    module = importlib.import_module(module_name)
  File "/home/raymondlo84/.local/share/uv/python/cpython-3.13.11-linux-aarch64-gnu/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raymondlo84/spark_serving/vllm/vllm/platforms/cuda.py", line 23, in <module>
    import vllm._C_stable_libtorch  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'vllm._C_stable_libtorch'

```

</details>


### 🐛 Describe the bug
(vllm) raymondlo84@spark-ray:~/spark_serving/vllm$ vllm serve nvidia/Gemma-4-26B-A4B-NVFP4   --kv-cache-dtype fp8   --gpu-memory-utilization 0.8   --max-num-seqs 8   --load-format fastsafetensors   --enable-prefix-caching   --max-num-batched-tokens 8192   --tensor-parallel-size 1   --enable-auto-tool-choice   --tool-call-parser gemma4   --chat-template examples/tool_chat_template_gemma4.jinja   --reasoning-parser gemma4   --speculative-config '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}'
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345] 
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.27.1
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345]   █▄█▀ █     █     █     █  model   nvidia/Gemma-4-26B-A4B-NVFP4
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:345] 
(APIServer pid=4108793) INFO 08-25 22:02:31 [api_utils.py:273] non-default args: {'model_tag': 'nvidia/Gemma-4-26B-A4B-NVFP4', 'chat_template': 'examples/tool_chat_template_gemma4.jinja', 'enable_auto_tool_choice': True, 'tool_call_parser': 'gemma4', 'model': 'nvidia/Gemma-4-26B-A4B-NVFP4', 'load_format': 'fastsafetensors', 'reasoning_parser': 'gemma4', 'gpu_memory_utilization': 0.8, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 8, 'speculative_config': {'model': 'google/gemma-4-26B-A4B-it-assistant', 'num_speculative_tokens': 4}}
(APIServer pid=4108793) Traceback (most recent call last):
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/bin/vllm", line 10, in <module>
(APIServer pid=4108793)     sys.exit(main())
(APIServer pid=4108793)              ~~~~^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/cli/main.py", line 97, in main
(APIServer pid=4108793)     args.dispatch_function(args)
(APIServer pid=4108793)     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/cli/serve.py", line 152, in cmd
(APIServer pid=4108793)     uvloop.run(run_server(args))
(APIServer pid=4108793)     ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=4108793)     return __asyncio.run(
(APIServer pid=4108793)            ~~~~~~~~~~~~~^
(APIServer pid=4108793)         wrapper(),
(APIServer pid=4108793)         ^^^^^^^^^^
(APIServer pid=4108793)     ...<2 lines>...
(APIServer pid=4108793)         **run_kwargs
(APIServer pid=4108793)         ^^^^^^^^^^^^
(APIServer pid=4108793)     )
(APIServer pid=4108793)     ^
(APIServer pid=4108793)   File "/home/raymondlo84/.local/share/uv/python/cpython-3.13.11-linux-aarch64-gnu/lib/python3.13/asyncio/runners.py", line 195, in run
(APIServer pid=4108793)     return runner.run(main)
(APIServer pid=4108793)            ~~~~~~~~~~^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/.local/share/uv/python/cpython-3.13.11-linux-aarch64-gnu/lib/python3.13/asyncio/runners.py", line 118, in run
(APIServer pid=4108793)     return self._loop.run_until_complete(task)
(APIServer pid=4108793)            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=4108793)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=4108793)     return await main
(APIServer pid=4108793)            ^^^^^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 764, in run_server
(APIServer pid=4108793)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 778, in run_server_worker
(APIServer pid=4108793)     async with build_async_engine_client(
(APIServer pid=4108793)                ~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=4108793)         args,
(APIServer pid=4108793)         ^^^^^
(APIServer pid=4108793)         client_config=client_config,
(APIServer pid=4108793)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)     ) as engine_client:
(APIServer pid=4108793)     ^
(APIServer pid=4108793)   File "/home/raymondlo84/.local/share/uv/python/cpython-3.13.11-linux-aarch64-gnu/lib/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=4108793)     return await anext(self.gen)
(APIServer pid=4108793)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 132, in build_async_engine_client
(APIServer pid=4108793)     async with build_async_engine_client_from_engine_args(
(APIServer pid=4108793)                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=4108793)         engine_args,
(APIServer pid=4108793)         ^^^^^^^^^^^^
(APIServer pid=4108793)         usage_context=usage_context,
(APIServer pid=4108793)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)         client_config=client_config,
(APIServer pid=4108793)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)     ) as engine:
(APIServer pid=4108793)     ^
(APIServer pid=4108793)   File "/home/raymondlo84/.local/share/uv/python/cpython-3.13.11-linux-aarch64-gnu/lib/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=4108793)     return await anext(self.gen)
(APIServer pid=4108793)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 156, in build_async_engine_client_from_engine_args
(APIServer pid=4108793)     vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/engine/arg_utils.py", line 1929, in create_engine_config
(APIServer pid=4108793)     model_config = self.create_model_config()
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/engine/arg_utils.py", line 1686, in create_model_config
(APIServer pid=4108793)     return ModelConfig(
(APIServer pid=4108793)         model=self.model,
(APIServer pid=4108793)     ...<63 lines>...
(APIServer pid=4108793)         renderer_num_workers=self.renderer_num_workers,
(APIServer pid=4108793)     )
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
(APIServer pid=4108793)     s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=4108793)     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/config/model.py", line 595, in __post_init__
(APIServer pid=4108793)     self.model_arch_config = self.get_model_arch_config()
(APIServer pid=4108793)                              ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/config/model.py", line 856, in get_model_arch_config
(APIServer pid=4108793)     return convertor.convert(
(APIServer pid=4108793)            ~~~~~~~~~~~~~~~~~^
(APIServer pid=4108793)         supports_multimodal=self._supports_multimodal_for_mm_prefix()
(APIServer pid=4108793)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4108793)     )
(APIServer pid=4108793)     ^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/transformers_utils/model_arch_config_convertor.py", line 378, in convert
(APIServer pid=4108793)     head_size=self.get_head_size(),
(APIServer pid=4108793)               ~~~~~~~~~~~~~~~~~~^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/vllm/transformers_utils/model_arch_config_convertor.py", line 608, in get_head_size
(APIServer pid=4108793)     head_dim = getattr(self.hf_text_config, "head_dim", 0)
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/transformers/configuration_utils.py", line 482, in __getattribute__
(APIServer pid=4108793)     return super().__getattribute__(key)
(APIServer pid=4108793)            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
(APIServer pid=4108793)   File "/home/raymondlo84/spark_serving/vllm/.venv/lib/python3.13/site-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 298, in __getattribute__
(APIServer pid=4108793)     raise AmbiguousGlobalPerLayerAttributeError(
(APIServer pid=4108793)     ...<5 lines>...
(APIServer pid=4108793)     )
(APIServer pid=4108793) transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.

The example is not working
https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it?variant=nvidia_nvfp4&hardware=dgx_spark_gb10&features=tool_calling%2Creasoning%2Cspec_decoding


vllm serve nvidia/Gemma-4-26B-A4B-NVFP4   --kv-cache-dtype fp8   --gpu-memory-utilization 0.8   --max-num-seqs 8   --load-format fastsafetensors   --enable-prefix-caching   --max-num-batched-tokens 8192   --tensor-parallel-size 1   --enable-auto-tool-choice   --tool-call-parser gemma4   --chat-template examples/tool_chat_template_gemma4.jinja   --reasoning-parser gemma4   --speculative-config '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}'

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

Navigation Menu

Uh oh!

Uh oh!

[Bug]: NVIDIA DGX Spark vllms and gemma 4 26B won't work #53836

Your current environment

🐛 Describe the bug

Before submitting a new issue...

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Uh oh!

[Bug]: NVIDIA DGX Spark vllms and gemma 4 26B won't work #53836

Description

Your current environment

🐛 Describe the bug

Before submitting a new issue...

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions