Skip to content

Commit fed0c45

Browse files
prekshivyasgithub-actions[bot]senthilr-nv
authored
feat(inference): add Linux amd64 Muse and Lightning profiles (#9537)
## Summary Add explicit-only Linux amd64 managed-vLLM profiles for Muse Glimmer 30B and NVIDIA Nemotron 3.5 Lightning 30B-A3B. This is the scoped experimental implementation authorized by #9673 and is stacked on the declarative catalog foundation in #9660. ## Changes - Add Linux x86_64 NVIDIA GPU presets for Muse and Lightning. - Add pinned amd64 vLLM recipes with model-specific images, memory floors, compute-capability floors, and serve arguments. - Keep both profiles explicit-only and experimental; no automatic support promotion or generic fallback is introduced. - Preserve the optimized DGX Spark recipes as the more specific selection. - Test catalog compilation, direct model selection, amd64/arm64 rejection, insufficient-memory rejection, runtime materialization, compute floors, and Spark precedence. ## Scope boundaries - No port configuration changes; vLLM port work remains in #9655. - No llama.cpp port or lifecycle changes; those remain separate in #9644. - No support-state promotion, default-model change, or arbitrary-model escape hatch. - No TypeScript model-specific routing branches; the additions are declarative catalog data. ## Product authority - Accepted implementation scope and ownership: #9673 - Declarative catalog foundation: #9660 - Related design direction: #7636 ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [x] Sensitive paths changed (inference selection and preflight) - [x] Sensitive-path review completed: selection remains explicit-only, images and revisions are pinned, architecture/memory/compute constraints are checked before materialization, and shipped recipes do not enable remote model code. - [ ] Non-success, skipped, or missing CI check accepted by maintainer ## Verification - [x] `npm run catalog:check` - [x] 67 focused compiler, resolver, and runtime-selection tests - [x] `npm run typecheck:cli` - [x] Focused Oxlint and formatting checks - [x] `npm run validate:pr` - [x] `git diff --check` ## Hardware evidence The earlier prototype completed managed onboarding for both models on a Brev RTX PRO Server 6000, including authenticated discovery, chat, tool calling, loopback binding, restart, and recovery. Current head `7d5d455f01db72919eab166450e060dedfbe3727` retains the validated runtime images and serve arguments; independent final-head validation remains pending. --- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Closes #9673 --------- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Senthil Ravichandran <senthilr@nvidia.com>
1 parent 289d5a5 commit fed0c45

8 files changed

Lines changed: 583 additions & 8 deletions

‎managed-inference/presets/vllm.linux-amd64-nvidia.single.muse-glimmer-30b-nvfp4-w4a4.yaml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: nemoclaw.nvidia.com/managed-inference/v1
5+
kind: ServingPreset
6+
7+
metadata:
8+
id: vllm.linux-amd64-nvidia.single.muse-glimmer-30b-nvfp4-w4a4
9+
displayName: Muse Glimmer 30B NVFP4 W4A4 on one Linux x86_64 NVIDIA GPU
10+
supportState: experimental
11+
validation:
12+
level: hardware
13+
evidence: brev-rtx-pro-server-6000-2026-08-18
14+
15+
spec:
16+
selection: explicit-only
17+
priority: 310
18+
19+
requirements:
20+
all:
21+
- readiness:
22+
scope: everyNode
23+
kind: capability
24+
id: host.platform.supported
25+
state: present
26+
- readiness:
27+
scope: everyNode
28+
kind: capability
29+
id: host.docker.available
30+
state: present
31+
- readiness:
32+
scope: everyNode
33+
kind: capability
34+
id: host.docker.daemon_reachable
35+
state: present
36+
- readiness:
37+
scope: everyNode
38+
kind: capability
39+
id: host.docker.runtime_supported
40+
state: present
41+
- readiness:
42+
scope: everyNode
43+
kind: capability
44+
id: host.docker.storage_compatible
45+
state: present
46+
- readiness:
47+
scope: everyNode
48+
kind: capability
49+
id: host.gpu.nvidia_available
50+
state: present
51+
- readiness:
52+
scope: everyNode
53+
kind: capability
54+
id: host.gpu.container_toolkit_available
55+
state: present
56+
- readiness:
57+
scope: everyNode
58+
kind: capability
59+
id: host.gpu.cdi_healthy
60+
state: present
61+
- readiness:
62+
scope: everyNode
63+
kind: observation
64+
id: host.os.platform
65+
comparison:
66+
operator: equals
67+
value: linux
68+
- readiness:
69+
scope: everyNode
70+
kind: observation
71+
id: host.os.architecture
72+
comparison:
73+
operator: equals
74+
value: x64
75+
- readiness:
76+
scope: everyNode
77+
kind: observation
78+
id: host.docker.runtime
79+
comparison:
80+
operator: equals
81+
value: docker
82+
- readiness:
83+
scope: everyNode
84+
kind: observation
85+
id: host.gpu.count
86+
comparison:
87+
operator: at-least
88+
value: 1
89+
- readiness:
90+
scope: everyNode
91+
kind: observation
92+
id: host.gpu.driver_version
93+
comparison:
94+
operator: version-at-least
95+
value: 580.65.06
96+
97+
plan:
98+
backend: vllm
99+
platform: linux
100+
interactive: false
101+
recipeRef: vllm.muse-glimmer-30b-nvfp4-w4a4.linux-amd64-single.v1

‎managed-inference/presets/vllm.linux-amd64-nvidia.single.nemotron-3.5-lightning-30b-a3b-nvfp4.yaml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: nemoclaw.nvidia.com/managed-inference/v1
5+
kind: ServingPreset
6+
7+
metadata:
8+
id: vllm.linux-amd64-nvidia.single.nemotron-3.5-lightning-30b-a3b-nvfp4
9+
displayName: NVIDIA Nemotron 3.5 Lightning 30B-A3B NVFP4 on one Linux x86_64 NVIDIA GPU
10+
supportState: experimental
11+
validation:
12+
level: hardware
13+
evidence: brev-rtx-pro-server-6000-2026-08-18
14+
15+
spec:
16+
selection: explicit-only
17+
priority: 300
18+
19+
requirements:
20+
all:
21+
- readiness:
22+
scope: everyNode
23+
kind: capability
24+
id: host.platform.supported
25+
state: present
26+
- readiness:
27+
scope: everyNode
28+
kind: capability
29+
id: host.docker.available
30+
state: present
31+
- readiness:
32+
scope: everyNode
33+
kind: capability
34+
id: host.docker.daemon_reachable
35+
state: present
36+
- readiness:
37+
scope: everyNode
38+
kind: capability
39+
id: host.docker.runtime_supported
40+
state: present
41+
- readiness:
42+
scope: everyNode
43+
kind: capability
44+
id: host.docker.storage_compatible
45+
state: present
46+
- readiness:
47+
scope: everyNode
48+
kind: capability
49+
id: host.gpu.nvidia_available
50+
state: present
51+
- readiness:
52+
scope: everyNode
53+
kind: capability
54+
id: host.gpu.container_toolkit_available
55+
state: present
56+
- readiness:
57+
scope: everyNode
58+
kind: capability
59+
id: host.gpu.cdi_healthy
60+
state: present
61+
- readiness:
62+
scope: everyNode
63+
kind: observation
64+
id: host.os.platform
65+
comparison:
66+
operator: equals
67+
value: linux
68+
- readiness:
69+
scope: everyNode
70+
kind: observation
71+
id: host.os.architecture
72+
comparison:
73+
operator: equals
74+
value: x64
75+
- readiness:
76+
scope: everyNode
77+
kind: observation
78+
id: host.docker.runtime
79+
comparison:
80+
operator: equals
81+
value: docker
82+
- readiness:
83+
scope: everyNode
84+
kind: observation
85+
id: host.gpu.count
86+
comparison:
87+
operator: at-least
88+
value: 1
89+
- readiness:
90+
scope: everyNode
91+
kind: observation
92+
id: host.gpu.driver_version
93+
comparison:
94+
operator: version-at-least
95+
value: 580.65.06
96+
97+
plan:
98+
backend: vllm
99+
platform: linux
100+
interactive: false
101+
recipeRef: vllm.nemotron-3.5-lightning-30b-a3b-nvfp4.linux-amd64-single.v1

‎managed-inference/recipes/vllm.muse-glimmer-30b-nvfp4-w4a4.linux-amd64-single.v1.yaml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: nemoclaw.nvidia.com/managed-inference/v1
5+
kind: ServingRecipe
6+
7+
metadata:
8+
id: vllm.muse-glimmer-30b-nvfp4-w4a4.linux-amd64-single.v1
9+
displayName: Muse Glimmer 30B NVFP4 W4A4 with vLLM on Linux x86_64
10+
11+
spec:
12+
backend: vllm
13+
14+
modelRef: vllm.muse-glimmer-30b-nvfp4-w4a4.v1
15+
runtime:
16+
image: vllm/vllm-openai@sha256:7eb4028507367e69cb0abfa213042d1814c27c1b499af45fbffec8f16d9cbc6f
17+
imageDownloadSizeBytes: 8632473449
18+
minimumComputeCapability: 120
19+
minimumGpuMemoryBytes: 96000000000
20+
pullTimeoutSeconds: 3600
21+
architecture: amd64
22+
networkMode: bridge
23+
ipcMode: host
24+
sharedMemoryBytes: 34359738368
25+
gpuRequest: all
26+
devices: []
27+
ulimits:
28+
memlock: -1
29+
stackBytes: 67108864
30+
modelCache:
31+
source: huggingface-cache
32+
target: /root/.cache/huggingface
33+
temporaryFilesystems: []
34+
environment: {}
35+
36+
execution:
37+
materializerRef: vllm.host-local/v1
38+
lifecycleRef: vllm.host-local.lifecycle/v1
39+
orchestrationRef: vllm.host-local.standard/v1
40+
41+
serve:
42+
authentication: bearer
43+
directInstall:
44+
authentication: bearer
45+
fixedArguments: true
46+
catalogReceipt: true
47+
executable: /usr/local/bin/vllm
48+
arguments:
49+
- name: --tokenizer
50+
value: /root/.cache/huggingface/hub/models--Inferact--Muse-Glimmer-30B-NVFP4-W4A4/snapshots/d35cb79050f419c457611b1cee5c5d15b176f285
51+
- name: --tokenizer-revision
52+
value: d35cb79050f419c457611b1cee5c5d15b176f285
53+
- name: --max-model-len
54+
value: 32768
55+
- name: --gpu-memory-utilization
56+
value: 0.75
57+
- name: --max-num-seqs
58+
value: 1
59+
- name: --max-num-batched-tokens
60+
value: 4096
61+
- name: --enable-auto-tool-choice
62+
- name: --tool-call-parser
63+
value: muse_glimmer
64+
- name: --reasoning-parser
65+
value: muse_glimmer
66+
- name: --generation-config
67+
value: auto
68+
69+
readiness:
70+
timeoutSeconds: 1800
71+
expectedModel: muse-glimmer

‎managed-inference/recipes/vllm.nemotron-3.5-lightning-30b-a3b-nvfp4.linux-amd64-single.v1.yaml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: nemoclaw.nvidia.com/managed-inference/v1
5+
kind: ServingRecipe
6+
7+
metadata:
8+
id: vllm.nemotron-3.5-lightning-30b-a3b-nvfp4.linux-amd64-single.v1
9+
displayName: NVIDIA Nemotron 3.5 Lightning 30B-A3B NVFP4 with vLLM on Linux x86_64
10+
11+
spec:
12+
backend: vllm
13+
14+
modelRef: vllm.nemotron-3.5-lightning-30b-a3b-nvfp4.v1
15+
runtime:
16+
# NVIDIA's published one-H100/Brev recipe uses vLLM v0.27.1. Pin the
17+
# Linux amd64 manifest rather than the multi-platform tag.
18+
image: vllm/vllm-openai@sha256:c2f3b1b964e47809b722b5e75b61b1e7b39a50f70388cf2bf2418f16a9f31da2
19+
imageDownloadSizeBytes: 9110652559
20+
# NVIDIA's published vLLM recipe matrix starts at H100. Lower compute
21+
# capabilities require separate qualification before they can use this profile.
22+
minimumComputeCapability: 90
23+
minimumGpuMemoryBytes: 96000000000
24+
pullTimeoutSeconds: 3600
25+
architecture: amd64
26+
networkMode: bridge
27+
ipcMode: host
28+
sharedMemoryBytes: 34359738368
29+
gpuRequest: all
30+
devices: []
31+
ulimits:
32+
memlock: -1
33+
stackBytes: 67108864
34+
modelCache:
35+
source: huggingface-cache
36+
target: /root/.cache/huggingface
37+
temporaryFilesystems: []
38+
environment: {}
39+
40+
execution:
41+
materializerRef: vllm.host-local/v1
42+
lifecycleRef: vllm.host-local.lifecycle/v1
43+
orchestrationRef: vllm.host-local.standard/v1
44+
45+
serve:
46+
authentication: bearer
47+
directInstall:
48+
authentication: bearer
49+
fixedArguments: true
50+
catalogReceipt: true
51+
executable: /usr/local/bin/vllm
52+
arguments:
53+
- name: --max-model-len
54+
value: 65536
55+
- name: --max-num-seqs
56+
value: 1
57+
- name: --gpu-memory-utilization
58+
value: 0.75
59+
- name: --max-num-batched-tokens
60+
value: 4096
61+
- name: --mamba-backend
62+
value: flashinfer
63+
- name: --enable-auto-tool-choice
64+
- name: --tool-call-parser
65+
value: qwen3_coder
66+
- name: --reasoning-parser
67+
value: nemotron_v3
68+
69+
readiness:
70+
timeoutSeconds: 1800
71+
expectedModel: nvidia-nemotron-3.5-lightning-30b-a3b-nvfp4

0 commit comments

Comments
 (0)