Skip to content

Commit def13e0

Browse files
feat(demos): credential-free NIM for CNCF AI conformance evidence (NVIDIA#2244)
Signed-off-by: Yuan Chen <yuanchen97@gmail.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com>
1 parent 9e951e4 commit def13e0

4 files changed

Lines changed: 173 additions & 3 deletions

File tree

demos/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,18 @@ Runbooks for testing and demonstrating AICR end-to-end workflows on live cluster
2828
| [private-signing.md](private-signing.md) | Private/enterprise signing & verification (self-hosted Sigstore, KMS-backed, headless OIDC) |
2929
| [examples/CUJ2-Test-Report.md](examples/CUJ2-Test-Report.md) | Dated historical capture (2026-03-13) of a CUJ2 inference run — example test report, not a runbook |
3030

31+
## Workload Samples
32+
33+
Deployable manifests used by the demos above and by conformance evidence collection. Each carries its prerequisites and setup commands in its header.
34+
35+
| Sample | Description |
36+
|--------|-------------|
37+
| [workloads/inference/vllm-agg.yaml](workloads/inference/vllm-agg.yaml) | Dynamo vLLM aggregated inference (DynamoGraphDeployment); pulls an ungated Hugging Face model, no credential required. **Caution:** also applies a cluster-scoped `Queue/dynamo` with `parentQueue: default-parent-queue` and zeroed quotas. Where `dynamo-platform` is installed it creates that same queue with `parentQueue: dynamo-default` and `quota: -1`, so applying this manifest silently repoints the platform's queue and rescopes every workload in it — and deleting the `dynamo-workload` namespace cannot revert a cluster-scoped object. Remove the `Queue` document from your copy of the manifest before applying on such a cluster (do not delete the live queue — it belongs to the platform). |
38+
| [workloads/inference/nimservice-llama-3-2-1b.yaml](workloads/inference/nimservice-llama-3-2-1b.yaml) | NIM inference via the NGC model path; requires two secrets as written — `ngc-api-secret` (`authSecret`, holding `NGC_API_KEY` for model download) and `ngc-pull-secret` (`image.pullSecrets`, for the `nvcr.io` pull) |
39+
| [workloads/inference/nimservice-hf-nocred.yaml](workloads/inference/nimservice-hf-nocred.yaml) | NIM inference via an `hf://` model; no NGC credential required (see [NIM workload credentials](../docs/user/component-catalog.md#nim-workload-credentials)) |
40+
| [workloads/inference/vllm-metrics-test.yaml](workloads/inference/vllm-metrics-test.yaml) | Standalone vLLM server with a Prometheus ServiceMonitor, used for AI Service Metrics evidence collection; no credential required |
41+
| [workloads/training/gke-nccl-test-tcpxo.yaml](workloads/training/gke-nccl-test-tcpxo.yaml) | NCCL all-reduce bandwidth test for GKE TCPXO fabric |
42+
3143
## Recording Test Runs
3244

3345
Use the `script` command to capture a terminal session for sharing or archival:
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# NIM Service — credential-free inference via a Hugging Face model.
16+
# Serves an ungated Hugging Face model through NIM without any NGC credential:
17+
# no NGC_API_KEY and no registry pull secret.
18+
#
19+
# How it avoids credentials:
20+
# - NIM_MODEL_NAME uses an hf:// URI, which puts the operator on its Hugging
21+
# Face path. There it marks NGC_API_KEY optional and injects HF_TOKEN from
22+
# the same authSecret, so the secret need not carry an NGC key at all.
23+
# - This repository (nim/meta/llama-3.1-8b-instruct) serves anonymous
24+
# registry tokens, so no pullSecrets are required. Availability is
25+
# per-repository, not namespace-wide: verify any other image before
26+
# assuming it pulls anonymously. The generic Multi-LLM image
27+
# nvcr.io/nim/nvidia/llm-nim is gated and does need a pull secret.
28+
#
29+
# spec.authSecret is required by the NIMService schema, so the secret must
30+
# exist and must contain an HF_TOKEN key. An empty value is sufficient for an
31+
# ungated model; a gated Hugging Face repository needs a real token.
32+
#
33+
# Off-label combination: this is a Llama-specific NIM image serving a Hugging
34+
# Face model it was not built for. The image keeps its own profile identity
35+
# (logs report model_tag meta/llama-3.1-8b-instruct) while vLLM serves the
36+
# downloaded Hugging Face weights. It works, but it is not the image's
37+
# advertised use, and "verified" below means the service loads and returns a
38+
# completion -- a liveness check. An off-label pairing can still apply the
39+
# wrong chat template or tokenizer and degrade output quality silently, so
40+
# spot-check generations before relying on this for anything but a smoke
41+
# test. The
42+
# image intended for arbitrary hf:// models is nvcr.io/nim/nvidia/llm-nim,
43+
# which is a gated repository and therefore needs an NGC pull secret --
44+
# choosing it trades the credential-free property for a supported pairing.
45+
# Pin the tag (not latest) so the pairing you validated is the one you ship.
46+
# The hf:// capability is version-dependent: 2.0.10 honours it, while 2.0.9
47+
# ignores NIM_MODEL_NAME, falls back to its own NGC profile, and fails with an
48+
# authentication error. Re-verify before moving this pin.
49+
#
50+
# Prerequisites:
51+
# - k8s-nim-operator deployed (via AICR NIM recipe); no credential needed
52+
#
53+
# Setup:
54+
# kubectl create ns nim-workload
55+
# kubectl create secret generic hf-secret \
56+
# --from-literal=HF_TOKEN="" -n nim-workload
57+
#
58+
# Deploy:
59+
# kubectl apply -f nimservice-hf-nocred.yaml
60+
#
61+
# Test:
62+
# kubectl port-forward svc/qwen3-nocred 8000:8000 -n nim-workload
63+
# curl http://localhost:8000/v1/models
64+
# curl http://localhost:8000/v1/chat/completions \
65+
# -H "Content-Type: application/json" \
66+
# -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello!"}],"max_tokens":30}'
67+
#
68+
# Node scheduling below targets an AKS GPU pool. Adjust nodeSelector and
69+
# tolerations for other platforms (EKS/GKE use different pool labels and taints).
70+
71+
apiVersion: v1
72+
kind: Namespace
73+
metadata:
74+
name: nim-workload
75+
---
76+
apiVersion: apps.nvidia.com/v1alpha1
77+
kind: NIMService
78+
metadata:
79+
name: qwen3-nocred
80+
namespace: nim-workload
81+
spec:
82+
# Required by the schema. Holds only HF_TOKEN — no NGC_API_KEY.
83+
authSecret: hf-secret
84+
image:
85+
repository: nvcr.io/nim/meta/llama-3.1-8b-instruct
86+
tag: "2.0.10"
87+
pullPolicy: IfNotPresent
88+
# No pullSecrets: this repository serves anonymous registry tokens.
89+
env:
90+
- name: NIM_MODEL_NAME
91+
value: hf://Qwen/Qwen3-0.6B
92+
- name: NIM_SERVED_MODEL_NAME
93+
value: Qwen/Qwen3-0.6B
94+
storage:
95+
emptyDir:
96+
sizeLimit: 60Gi
97+
replicas: 1
98+
resources:
99+
limits:
100+
nvidia.com/gpu: 1
101+
nodeSelector:
102+
# Replace gpuworker1 with your own GPU pool's agentpool label.
103+
agentpool: gpuworker1
104+
tolerations:
105+
- key: nvidia.com/gpu
106+
operator: Equal
107+
value: present
108+
effect: NoSchedule
109+
expose:
110+
service:
111+
type: ClusterIP
112+
port: 8000

demos/workloads/inference/nimservice-llama-3-2-1b.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@
2424
# kubectl create ns nim-workload
2525
# kubectl create secret docker-registry ngc-pull-secret \
2626
# --docker-server=nvcr.io --docker-username='$oauthtoken' \
27-
# --docker-password="$NGC_CLI_API_KEY" -n nim-workload
27+
# --docker-password="$NGC_API_KEY" -n nim-workload
2828
# kubectl create secret generic ngc-api-secret \
29-
# --from-literal=NGC_API_KEY="$NGC_CLI_API_KEY" -n nim-workload
29+
# --from-literal=NGC_API_KEY="$NGC_API_KEY" -n nim-workload
3030
#
3131
# Deploy:
3232
# kubectl apply -f nimservice-llama-3-2-1b.yaml

docs/user/component-catalog.md

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
3333
| **dynamo-platform** | NVIDIA Dynamo inference serving platform with bundled CRDs. Distributed inference with KV-cache-aware routing, Dynamo request-plane traffic, a NATS-backed Kubernetes event plane for KV-cache events, and disaggregated prefill/decode. | [Dynamo](https://github.com/ai-dynamo/dynamo) |
3434
| **agentgateway-crds** | Custom Resource Definitions for agentgateway (Kubernetes Gateway API implementation for AI/ML inference). | [agentgateway](https://github.com/agentgateway/agentgateway) |
3535
| **agentgateway** | Kubernetes Gateway API implementation for AI/ML inference. Implements the Gateway API Inference Extension for model-aware ingress routing to InferencePool backends. | [agentgateway](https://github.com/agentgateway/agentgateway) |
36-
| **k8s-nim-operator** | NVIDIA NIM Operator for managing NIM (NVIDIA Inference Microservices) deployments on Kubernetes. | [K8s NIM Operator](https://github.com/NVIDIA/k8s-nim-operator) |
36+
| **k8s-nim-operator** | NVIDIA NIM Operator for managing NIM (NVIDIA Inference Microservices) deployments on Kubernetes. AICR installs the operator only — it creates no `NIMService` and no credentials; see [NIM workload credentials](#nim-workload-credentials). | [K8s NIM Operator](https://github.com/NVIDIA/k8s-nim-operator) |
3737
| **kueue** | Kubernetes-native job queuing system. Manages quotas and admits jobs for batch and AI workloads. Ships default quota CRs (ResourceFlavor `default-flavor`, ClusterQueue `cluster-queue`, LocalQueue `default` in the `default` namespace) so admission works out of the box — tune the ClusterQueue's nominal quotas to cluster capacity to enact real limits. Managed frameworks are pinned to batch/job, JobSet, and TrainJob. Upgrade note: the quota CRs are helm post-install/post-upgrade hooks with a delete-and-recreate policy — quiesce queues before upgrading the bundle (Kueue's resource-in-use finalizer on an active ClusterQueue/ResourceFlavor blocks the delete and can wedge the upgrade), and re-apply tuned quotas afterwards since upgrades reset them to the shipped defaults. Uninstalling leaves the hook-created CRs behind; delete them manually when removing Kueue. Overlays that override the component's `manifestFiles` (replacing the default quota CRs) must also override its health check — the shipped check asserts the default CR names above. | [Kueue](https://github.com/kubernetes-sigs/kueue) |
3838
| **kubeflow-trainer** | Kubeflow Training Operator for distributed training jobs (PyTorch, etc.). Manages multi-node training job lifecycle with JobSet integration. | [Kubeflow Trainer](https://github.com/kubeflow/trainer) |
3939
| **mariadb-operator-crds** | Official MariaDB Operator CRDs. Declared in every Slurm recipe but installed only for `accounting.mode: aicr-provided`. | [MariaDB Operator](https://github.com/mariadb-operator/mariadb-operator) |
@@ -197,6 +197,52 @@ aicr bundle -r recipe.yaml \
197197

198198
See [AKS GPU Setup](../integrator/aks-gpu-setup.md#default-use-the-aks-azure-managed-profile) for the per-profile guidance.
199199

200+
## NIM Workload Credentials
201+
202+
AICR installs the **k8s-nim-operator** only. It does not create a `NIMService` and does not create credentials — deploying a workload is an operator step, and there are two ways to supply the model.
203+
204+
Whichever path you take, `spec.authSecret` is required by the `NIMService` schema and must name an existing secret in the workload's namespace. `spec.image.pullSecrets` is optional; the image block requires only `repository` and `tag`.
205+
206+
### NGC path
207+
208+
Model artifacts come from NGC, so the secret must carry a valid `NGC_API_KEY`:
209+
210+
```bash
211+
kubectl create secret generic ngc-api-secret \
212+
--from-literal=NGC_API_KEY="$NGC_API_KEY" -n nim-workload
213+
```
214+
215+
Add a `docker-registry` secret and reference it from `image.pullSecrets` when the image lives in a private or authenticated registry path. See `demos/workloads/inference/nimservice-llama-3-2-1b.yaml` for a complete example.
216+
217+
### Credential-free path (Hugging Face)
218+
219+
Setting `NIM_MODEL_NAME` to an `hf://` URI puts the operator on its Hugging Face path, where it marks `NGC_API_KEY` optional and injects `HF_TOKEN` from the same `authSecret`. With an ungated Hugging Face model and a NIM image that pulls anonymously, no NGC credential is needed anywhere:
220+
221+
```bash
222+
kubectl create secret generic hf-secret --from-literal=HF_TOKEN="" -n nim-workload
223+
```
224+
225+
```yaml
226+
spec:
227+
authSecret: hf-secret # holds only HF_TOKEN
228+
image:
229+
repository: nvcr.io/nim/meta/llama-3.1-8b-instruct # pulls anonymously; no pullSecrets
230+
tag: "2.0.10" # pin a version; avoid the mutable latest
231+
env:
232+
- name: NIM_MODEL_NAME
233+
value: hf://Qwen/Qwen3-0.6B # ungated model
234+
- name: NIM_SERVED_MODEL_NAME
235+
value: Qwen/Qwen3-0.6B # the OpenAI-API `model` id
236+
```
237+
238+
The `HF_TOKEN` key must exist in the secret — that reference is not optional — but an empty value is sufficient for an ungated model. A gated Hugging Face repository needs a real token here.
239+
240+
Model-specific NIM repositories (for example `nim/meta/llama-3.1-8b-instruct`) serve anonymous registry tokens; the generic Multi-LLM image `nim/nvidia/llm-nim` does not and requires a pull secret.
241+
242+
Note that pairing a model-specific image with an unrelated `hf://` model is off-label: the container runs its own profile against the downloaded weights. It works, but `nim/nvidia/llm-nim` is the image intended for arbitrary Hugging Face models — and because that repository is gated, choosing it trades the credential-free property for a supported pairing. Pin an image tag rather than `latest` so the pairing you validated is the one you ship.
243+
244+
See `demos/workloads/inference/nimservice-hf-nocred.yaml` for a complete example.
245+
200246
## Inference Gateway Network Exposure
201247

202248
Inference recipes include the **agentgateway** component, which deploys an `inference-gateway` Gateway. The agentgateway controller materializes that Gateway into a `Service` of type `LoadBalancer`, so on every cloud the platform provisions a load balancer for the (plaintext HTTP, unauthenticated) inference endpoint. Left unrestricted that load balancer is internet-facing, so `aicr bundle` scopes it to private networks by default — the opt-in path for public exposure and the validation behavior are described below.

0 commit comments

Comments
 (0)