You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Addresses all open review threads from the superseded PR:
- Topograph now targets the chart-owned slinky-slurm-config-extra ConfigMap
(patched key-level), seeded via configFiles so topology.conf is mounted
into slurmctld from first boot; TopologyPlugin=topology/tree set on both
topograph-carrying leaves. Deploy order flipped (slinky-slurm before
slinky-topograph) so Helm owns the ConfigMap.
- Health check is assert/error-only (read-only allowlist); the Kind leaf
carries an inline healthCheckAsserts override (no DaemonSet step, exact
test-provider topology assertion).
- Docs and BOM synced.
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Copy file name to clipboardExpand all lines: demos/cuj1-slinky-slurm.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ Slurm leaves are built from criteria flags (`--service`, `--platform slurm`, …
26
26
27
27
## Generate Recipe (Query Mode)
28
28
29
-
Pick the row that matches your cluster. Each resolves to a slurm leaf with at least three inline Slinky components: `slinky-slurm-operator-crds`, `slinky-slurm-operator`, and `slinky-slurm`. The Kind leaf also includes`slinky-topograph` (topology-aware scheduling, test provider); cloud leaves can opt in to `slinky-topograph` with a real cloud provider.
29
+
Pick the row that matches your cluster. Each resolves to a slurm leaf with at least three inline Slinky components: `slinky-slurm-operator-crds`, `slinky-slurm-operator`, and `slinky-slurm`. The GKE and Kind leaves also include`slinky-topograph` (topology-aware scheduling) — GKE with the `gcp` provider, Kind with the `test` provider and a fixed topology fixture. The EKS leaf does not include it today; see [Slinky Slurm Inline Components](../docs/integrator/recipe-development.md#slinky-slurm-inline-components) to add it to another leaf.
30
30
31
31
32
32
| Cloud | Command | Leaf overlay |
@@ -135,7 +135,7 @@ Set `--storage-class` to a StorageClass that exists (`kubectl get storageclass`)
Deploy order: `cert-manager` → `slinky-slurm-operator-crds` → `slinky-slurm-operator` → `slinky-slurm` (→ `slinky-topograph` on the GKE and Kind leaves, after `slinky-slurm` so the slurm chart owns the ConfigMap Topograph patches).
139
139
140
140
```shell
141
141
kubectl rollout status -n slinky deploy/slurm-operator
Copy file name to clipboardExpand all lines: docs/integrator/recipe-development.md
+47-37Lines changed: 47 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -125,9 +125,43 @@ For example, `--platform slurm` leaves inline three `componentRefs`:
125
125
126
126
This is the same shape `dynamo-platform` uses across the `*-inference-dynamo` leaves. See `recipes/overlays/h100-eks-ubuntu-training-slurm.yaml` for the full example.
127
127
128
+
When authoring a recipe targeting Talos (`criteria.os: talos`), append the `os-talos` mixin to your overlay's `spec.mixins` list (e.g. `spec.mixins: [os-talos]`, or `[platform-kubeflow, os-talos]` if you already mix in a non-OS fragment). OS-scoped mixins are mutually exclusive — combining `os-ubuntu` and `os-talos` in one overlay is a recipe authoring error, not a supported composition. The mixin overrides namespaces for affected components and supplies PSA-privileged Namespace manifests via `componentRefs[].preManifestFiles`, which are applied before each chart — see [Talos integration](talos-integration.md) for the component list and labels.
129
+
130
+
**Cross-cutting overlays with wildcard criteria** apply across one criteria dimension without being referenced via `spec.base` or listed in `spec.mixins`. The resolver can return multiple independent maximal-leaf overlays for a single query, so a `service: any` overlay is picked up alongside the service-specific maximal leaf and its inheritance chain:
131
+
132
+
```yaml
133
+
# gb200-any.yaml — applies to every GB200 query (any service, any intent)
134
+
spec:
135
+
base: base
136
+
criteria:
137
+
service: any # Wildcard — matches eks, oke, gke, etc.
138
+
accelerator: gb200
139
+
validation:
140
+
deployment:
141
+
checks:
142
+
- operator-health
143
+
- expected-resources
144
+
- gpu-operator-version
145
+
- check-nvidia-smi
146
+
constraints:
147
+
- name: Deployment.gpu-operator.version
148
+
value: ">= v25.10.0"
149
+
```
150
+
151
+
Only use this pattern when the content is truly uniform across the wildcard dimension — if values diverge per service, keep them inline in each service-specific overlay. NCCL performance thresholds, for example, are explicitly **not** a good fit for this pattern: each service has a different network fabric (EFA, TCPXO, RoCE, etc.) and the same bandwidth number is rarely correct across two fabrics. The intent-scoped `gb200-any-training.yaml` and `b200-any-training.yaml` shapes that previously carried cross-service NCCL thresholds were retired (`gb200-any-training` in #1052, `b200-any-training` in #1053) in favor of per-leaf performance blocks. See [Data Architecture](../contributor/recipe.md#criteria-wildcard-overlays) for when to use wildcard overlays vs mixins.
- ComponentRefs: same-named merged field-by-field, new added
158
+
- `validation.<phase>` blocks merge per-field: `checks`and `constraints` union and deduplicate when non-empty (`constraints` by name, overlay wins on same-name); an explicit empty list (`checks: []` / `constraints: []`) **clears** the inherited list, while an omitted/null field **inherits** it; `nodeSelection` replaced wholesale when set; `timeout`/`infrastructure` overlay-wins-if-non-empty
159
+
- Criteria: not inherited (each recipe defines its own)
160
+
- Mixin constraints/components must not conflict with the inheritance chain or other mixins
161
+
128
162
### Slinky Slurm Inline Components
129
163
130
-
Leaves that need topology-aware scheduling can optionally add `slinky-topograph` as a fourth `componentRef`. It must be placed between `slinky-slurm-operator` and `slinky-slurm`, with `slinky-slurm` declaring it as a `dependencyRef`. The `overrides` block supplies the provider and engine that are specific to each leaf:
164
+
Leaves that need topology-aware scheduling can optionally add `slinky-topograph` as a fourth `componentRef`. Its `dependencyRefs` include `slinky-slurm`, so it deploys **after** the Slurm cluster chart: `slinky-slurm`renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first, or a later `helm upgrade` on `slinky-slurm` would fight Topograph for ownership. The `overrides` block supplies the provider and engine that are specific to each leaf:
131
165
132
166
```yaml
133
167
- name: slinky-topograph
@@ -136,6 +170,7 @@ Leaves that need topology-aware scheduling can optionally add `slinky-topograph`
136
170
dependencyRefs:
137
171
- slinky-slurm-operator
138
172
- slinky-slurm-operator-crds
173
+
- slinky-slurm # deploy after Slurm so Helm already owns config-extra
139
174
overrides:
140
175
global:
141
176
provider:
@@ -149,8 +184,17 @@ Leaves that need topology-aware scheduling can optionally add `slinky-topograph`
149
184
dependencyRefs:
150
185
- slinky-slurm-operator
151
186
- slinky-slurm-operator-crds
152
-
- slinky-topograph # ensures topograph is ready before Slurm starts
153
187
overrides:
188
+
configFiles:
189
+
# Seed so topology.conf exists in slinky-slurm-config-extra from first
190
+
# boot; Topograph overwrites this key after each successful sync.
191
+
# NOTE: helm upgrade resets the key to this seed until the next
192
+
# Topograph sync.
193
+
topology.conf: |
194
+
# Managed by NVIDIA Topograph (engine: slinky). Pre-sync seed.
For `provider.name: dra` (Kubernetes Dynamic Resource Allocation, GA in K8s 1.34), topology is sourced from the DRA API — no cloud provider IAM or ServiceAccount annotations are needed. Use `dra` for local or non-cloud environments such as Kind-based CI clusters.
178
-
179
-
When authoring a recipe targeting Talos (`criteria.os: talos`), append the `os-talos` mixin to your overlay's `spec.mixins` list (e.g. `spec.mixins: [os-talos]`, or `[platform-kubeflow, os-talos]` if you already mix in a non-OS fragment). OS-scoped mixins are mutually exclusive — combining `os-ubuntu` and `os-talos` in one overlay is a recipe authoring error, not a supported composition. The mixin overrides namespaces for affected components and supplies PSA-privileged Namespace manifests via `componentRefs[].preManifestFiles`, which are applied before each chart — see [Talos integration](talos-integration.md) for the component list and labels.
180
-
181
-
**Cross-cutting overlays with wildcard criteria** apply across one criteria dimension without being referenced via `spec.base` or listed in `spec.mixins`. The resolver can return multiple independent maximal-leaf overlays for a single query, so a `service: any` overlay is picked up alongside the service-specific maximal leaf and its inheritance chain:
182
-
183
-
```yaml
184
-
# gb200-any.yaml — applies to every GB200 query (any service, any intent)
185
-
spec:
186
-
base: base
187
-
criteria:
188
-
service: any # Wildcard — matches eks, oke, gke, etc.
189
-
accelerator: gb200
190
-
validation:
191
-
deployment:
192
-
checks:
193
-
- operator-health
194
-
- expected-resources
195
-
- gpu-operator-version
196
-
- check-nvidia-smi
197
-
constraints:
198
-
- name: Deployment.gpu-operator.version
199
-
value: ">= v25.10.0"
200
-
```
201
-
202
-
Only use this pattern when the content is truly uniform across the wildcard dimension — if values diverge per service, keep them inline in each service-specific overlay. NCCL performance thresholds, for example, are explicitly **not** a good fit for this pattern: each service has a different network fabric (EFA, TCPXO, RoCE, etc.) and the same bandwidth number is rarely correct across two fabrics. The intent-scoped `gb200-any-training.yaml` and `b200-any-training.yaml` shapes that previously carried cross-service NCCL thresholds were retired (`gb200-any-training` in #1052, `b200-any-training` in #1053) in favor of per-leaf performance blocks. See [Data Architecture](../contributor/recipe.md#criteria-wildcard-overlays) for when to use wildcard overlays vs mixins.
- ComponentRefs: same-named merged field-by-field, new added
209
-
- `validation.<phase>` blocks merge per-field: `checks`and `constraints` union and deduplicate when non-empty (`constraints` by name, overlay wins on same-name); an explicit empty list (`checks: []` / `constraints: []`) **clears** the inherited list, while an omitted/null field **inherits** it; `nodeSelection` replaced wholesale when set; `timeout`/`infrastructure` overlay-wins-if-non-empty
210
-
- Criteria: not inherited (each recipe defines its own)
211
-
- Mixin constraints/components must not conflict with the inheritance chain or other mixins
221
+
For `provider.name: dra` (Kubernetes Dynamic Resource Allocation, GA in K8s 1.34), topology is sourced from the DRA API — no cloud provider IAM or ServiceAccount annotations are needed. Use `dra` for clusters actually running DRA drivers with GPU resource claims. Kind-based CI clusters should use the `test` provider with a model fixture instead (as this repo's `h100-kind-training-slurm` overlay does) — a CPU-only Kind cluster has no DRA resources for the `dra` provider to read.
Copy file name to clipboardExpand all lines: docs/user/component-catalog.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,7 +39,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
39
39
|**slinky-slurm-operator-crds**| Custom Resource Definitions for the SchedMD Slinky Slurm operator. Installs the `slinky.slurm.net` CRDs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). Installed separately to support CRD lifecycle management. |[Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator)|
40
40
|**slinky-slurm-operator**| SchedMD Slinky Slurm operator and admission webhook. Manages the lifecycle of Slurm clusters declared via Slinky CRs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). AICR's system node-selector and toleration bundle flags apply to both deployments; affinity remains available through component values or typed overrides. |[Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator)|
41
41
|**slinky-slurm**| Slinky-managed Slurm cluster instance: Controller (slurmctld) + LoginSet (sackd/sshd) + NodeSet (slurmd) + RestApi (slurmrestd). Reconciled by `slinky-slurm-operator`. Declared inline per slurm leaf overlay alongside `slinky-slurm-operator-crds` and `slinky-slurm-operator` (matching the dynamo-platform pattern) so each leaf can carry its own GPU/GRES tuning. IMEX-capable leaves attach a fixed NVIDIA DRA `ComputeDomain` as a pre-manifest before the Slurm chart; the DRA driver reconciles it asynchronously into the `ResourceClaimTemplate` consumed by the NodeSet. Accounting (slurmdbd) requires an external MariaDB and is disabled in defaults — see `recipes/components/slinky-slurm/values.yaml`. |[Slinky Slurm Cluster Chart](https://github.com/SlinkyProject/slurm-operator/tree/main/helm/slurm)|
42
-
|**slinky-topograph**| Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm `topology.conf`, enabling topology-aware placement decisions in the Slinky-managed scheduler. **Not installed by default**; leaf overlays opt in by (1) adding an explicit `componentRef` entry for `slinky-topograph` and (2) listing it in `slinky-slurm`'s `dependencyRefs` to enforce install ordering. `dependencyRefs` alone is not sufficient — the `componentRef` is what schedules the release. Installed after `slinky-slurm-operator` and before `slinky-slurm` so the topology config is available when the cluster instance starts. Includes the `node-observer` sub-chart, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP `roles/compute.viewer` for Workload Identity). |[Topograph](https://github.com/NVIDIA/topograph)|
42
+
| **slinky-topograph** | Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm `topology.conf`, enabling topology-aware placement decisions in the Slinky-managed scheduler. **Not installed by default**; leaf overlays opt in by adding an explicit `componentRef` entry for `slinky-topograph` — the `componentRef` is what schedules the release; `dependencyRefs` alone does not install anything. That `componentRef` declares `slinky-slurm` as a `dependencyRef` to deploy **after** it: `slinky-slurm` renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first. `TopologyPlugin: topology/tree` is set per-leaf via `slinky-slurm`'s `controller.extraConfMap`. Includes the `node-observer` sub-chart, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP `roles/compute.viewer` for Workload Identity). | [Topograph](https://github.com/NVIDIA/topograph) |
43
43
|**nfd-ocp-olm**| OLM installer for Node Feature Discovery on OpenShift. Creates the OperatorGroup and Subscription resources that install NFD via the Operator Lifecycle Manager. Paired with `nfd-ocp`. OCP-specific. |[Node Feature Discovery (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5ec53e8c110f56bd24f5f8db)|
44
44
|**nfd-ocp**| Node Feature Discovery CR for OpenShift. Configures NFD's operand (worker, topology updater) via a NodeFeatureDiscovery custom resource. Deployed after `nfd-ocp-olm`. OCP-specific. |[Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery)|
45
45
|**gpu-operator-ocp-olm**| OLM installer for the GPU Operator on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified GPU Operator via the Operator Lifecycle Manager. Paired with `gpu-operator-ocp`. OCP-specific. |[NVIDIA GPU Operator (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5e7b210b8a3c1e00013d636d)|
@@ -55,7 +55,7 @@ Not every component appears in every recipe. The recipe engine selects component
55
55
-**Cloud-specific components** (aws-efa, aws-ebs-csi-driver) are added when the service matches. OCP recipes replace base components (gpu-operator, nfd, network-operator) with OLM+CR pairs (e.g., `gpu-operator-ocp-olm` + `gpu-operator-ocp`).
56
56
-**Intent-specific components** (agentgateway, agentgateway-crds) are added based on workload intent (e.g., inference recipes include the inference gateway).
57
57
-**Platform-specific components** (slinky-slurm-operator, slinky-slurm, kubeflow-trainer, dynamo-platform) are added when the recipe selects a matching `--platform`. For `--platform slurm`, all three core Slinky pieces (`slinky-slurm-operator-crds`, `slinky-slurm-operator`, `slinky-slurm`) are declared inline per slurm leaf overlay — the same shape `dynamo-platform` uses across `*-inference-dynamo` leaves. IMEX-capable Slurm leaves attach a fixed ComputeDomain through `slinky-slurm.preManifestFiles` so slurmd pods can consume DRA-provisioned IMEX channels. Leaves that want the operator only inline the CRDs + operator and omit the `slinky-slurm` componentRef. For an end-to-end walkthrough (recipe → bundle → install → validate → `srun` smoke job on EKS, GKE, or Kind), see [`demos/cuj1-slinky-slurm.md`](https://github.com/NVIDIA/aicr/blob/main/demos/cuj1-slinky-slurm.md).
58
-
-**Topology-aware optional components** (`slinky-topograph`) are not installed by default. Opting in requires two things in the leaf overlay: an explicit `componentRef` entry for `slinky-topograph`(which installs it) and a `dependencyRef` on`slinky-slurm`pointing to it (which enforces ordering). `dependencyRefs` alone does not install the component. See the wiring example in the [Recipe Development Guide](../integrator/recipe-development.md#slinky-slurm-inline-components).
58
+
-**Topology-aware optional components** (`slinky-topograph`) are not installed by default. Opting in requires an explicit `componentRef` entry for `slinky-topograph`in the leaf overlay — the `componentRef` is what installs it; `dependencyRefs` alone does not. That `componentRef` declares`slinky-slurm`as a `dependencyRef`, so Topograph deploys after the Slurm cluster chart, which owns the ConfigMap Topograph patches. See the wiring example in the [Recipe Development Guide](../integrator/recipe-development.md#slinky-slurm-inline-components).
59
59
-**Accelerator/OS-specific tuning** (nodewright-customizations, nvidia-dra-driver-gpu) varies by hardware and OS combination.
0 commit comments