feat(slinky-topograph): topograph component with slinky engine (NVIDIA#1737)

Signed-off-by: Ravi Shankar <ravish@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Co-authored-by: Ravi Shankar <ravish@nvidia.com>

Author: ArangoGutierrez

demos/cuj1-slinky-slurm.md

@@ -26,7 +26,7 @@ Slurm leaves are built from criteria flags (`--service`, `--platform slurm`, …
 
 ## Generate Recipe (Query Mode)
 
-Pick the row that matches your cluster. Each resolves to a slurm leaf with three inline Slinky components: `slinky-slurm-operator-crds`, `slinky-slurm-operator`, and `slinky-slurm`.
+Pick the row that matches your cluster. Each resolves to a slurm leaf with at least three inline Slinky components: `slinky-slurm-operator-crds`, `slinky-slurm-operator`, and `slinky-slurm`. The GKE and Kind leaves also include `slinky-topograph` (topology-aware scheduling) — GKE with the `gcp` provider, Kind with the `test` provider and a fixed topology fixture. The EKS leaf does not include it today; see [Slinky Slurm Inline Components](../docs/integrator/recipe-development.md#slinky-slurm-inline-components) to add it to another leaf.
 
 
 | Cloud    | Command                                                                                                      | Leaf overlay                                               |
@@ -135,7 +135,7 @@ Set `--storage-class` to a StorageClass that exists (`kubectl get storageclass`)
 cd ./bundle && chmod +x deploy.sh && ./deploy.sh
 ```
 
-Deploy order: `cert-manager` → `slinky-slurm-operator-crds` → `slinky-slurm-operator` → `slinky-slurm`.
+Deploy order: `cert-manager` → `slinky-slurm-operator-crds` → `slinky-slurm-operator` → `slinky-slurm` (→ `slinky-topograph` on the GKE and Kind leaves, after `slinky-slurm` so the slurm chart owns the ConfigMap Topograph patches).
 
 ```shell
 kubectl rollout status -n slinky deploy/slurm-operator
@@ -257,7 +257,7 @@ Cluster instance only (keep operator + CRDs):
 helm uninstall slinky-slurm -n slurm
 ```
 
-Full Slurm stack:
+Full Slurm stack (without topograph):
 
 ```shell
 helm uninstall slinky-slurm -n slurm
@@ -266,6 +266,16 @@ helm uninstall slinky-slurm-operator-crds -n slinky
 kubectl delete ns slurm slinky --ignore-not-found
 ```
 
+Full Slurm stack with topograph:
+
+```shell
+helm uninstall slinky-topograph -n topograph
+helm uninstall slinky-slurm -n slurm
+helm uninstall slinky-slurm-operator -n slinky
+helm uninstall slinky-slurm-operator-crds -n slinky
+kubectl delete ns slurm topograph slinky --ignore-not-found
+```
+
 Helm does not remove CRDs or PVCs by default; delete manually when you need a clean re-install.
 
 ## Success

docs/integrator/recipe-development.md

@@ -159,6 +159,71 @@ Only use this pattern when the content is truly uniform across the wildcard dime
 - Criteria: not inherited (each recipe defines its own)
 - Mixin constraints/components must not conflict with the inheritance chain or other mixins
 
+### Slinky Slurm Inline Components
+
+Leaves that need topology-aware scheduling can optionally add `slinky-topograph` as a fourth `componentRef`. Its `dependencyRefs` include `slinky-slurm`, so it deploys **after** the Slurm cluster chart: `slinky-slurm` renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first, or a later `helm upgrade` on `slinky-slurm` would fight Topograph for ownership. The `overrides` block supplies the provider and engine that are specific to each leaf:
+
+```yaml
+- name: slinky-topograph
+  type: Helm
+  valuesFile: components/slinky-topograph/values.yaml
+  dependencyRefs:
+    - slinky-slurm-operator
+    - slinky-slurm-operator-crds
+    - slinky-slurm   # deploy after Slurm so Helm already owns config-extra
+  overrides:
+    global:
+      provider:
+        name: gcp      # cloud provider: gcp | aws | oci | nebius | …
+      engine:
+        name: slinky   # scheduler consumer: slinky | slurm | k8s | graph
+
+- name: slinky-slurm
+  type: Helm
+  valuesFile: components/slinky-slurm/values.yaml
+  dependencyRefs:
+    - slinky-slurm-operator
+    - slinky-slurm-operator-crds
+  overrides:
+    configFiles:
+      # Seed so topology.conf exists in slinky-slurm-config-extra from first
+      # boot; Topograph overwrites this key after each successful sync.
+      # NOTE: helm upgrade resets the key to this seed until the next
+      # Topograph sync.
+      topology.conf: |
+        # Managed by NVIDIA Topograph (engine: slinky). Pre-sync placeholder.
+        SwitchName=aicr-preseed Nodes=aicr-preseed-node
+    controller:
+      extraConfMap:
+        TopologyPlugin: "topology/tree"
+      # Reload path: the reconfigure sidecar loads Topograph's topology.conf
+      # updates into the running slurmctld (rollout hash excludes configFileRefs).
+      inplaceReconfigure: true
+    # ... GPU GRES and other leaf-specific tuning ...
+```
+
+For cloud providers (gcp, aws, oci, nebius, …), `slinky-topograph` requires IAM access to call the cloud's topology API. For GKE, bind a GCP service account that has `roles/compute.viewer` on the project via Workload Identity:
+
+```yaml
+overrides:
+  global:
+    provider:
+      name: gcp
+  serviceAccount:
+    annotations:
+      iam.gke.io/gcp-service-account: <sa-name>@<project-id>.iam.gserviceaccount.com
+```
+
+If you prefer not to bake the GCP service account into the recipe, supply it at bundle time instead:
+
+```bash
+aicr bundle --recipe recipe.yaml \
+  --set-json 'slinkytopograph:serviceAccount.annotations={"iam.gke.io/gcp-service-account":"<sa-name>@<project-id>.iam.gserviceaccount.com"}' \
+  -o ./bundle
+```
+
+For `provider.name: dra` (Kubernetes Dynamic Resource Allocation, GA in K8s 1.34), topology is sourced from the DRA API — no cloud provider IAM or ServiceAccount annotations are needed. Use `dra` for clusters actually running DRA drivers with GPU resource claims. Kind-based CI clusters should use the `test` provider with a model fixture instead (as this repo's `h100-kind-training-slurm` overlay does) — a CPU-only Kind cluster has no DRA resources for the `dra` provider to read.
+
 ### Inference performance constraints
 
 The `inference-perf` performance check reads named entries from

docs/user/component-catalog.md

@@ -39,6 +39,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
 | **slinky-slurm-operator-crds** | Custom Resource Definitions for the SchedMD Slinky Slurm operator. Installs the `slinky.slurm.net` CRDs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). Installed separately to support CRD lifecycle management. | [Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator) |
 | **slinky-slurm-operator** | SchedMD Slinky Slurm operator and admission webhook. Manages the lifecycle of Slurm clusters declared via Slinky CRs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). AICR's system node-selector and toleration bundle flags apply to both deployments; affinity remains available through component values or typed overrides. | [Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator) |
 | **slinky-slurm** | Slinky-managed Slurm cluster instance: Controller (slurmctld) + LoginSet (sackd/sshd) + NodeSet (slurmd) + RestApi (slurmrestd). Reconciled by `slinky-slurm-operator`. Declared inline per slurm leaf overlay alongside `slinky-slurm-operator-crds` and `slinky-slurm-operator` (matching the dynamo-platform pattern) so each leaf can carry its own GPU/GRES tuning. IMEX-capable leaves attach a fixed NVIDIA DRA `ComputeDomain` as a pre-manifest before the Slurm chart; the DRA driver reconciles it asynchronously into the `ResourceClaimTemplate` consumed by the NodeSet. Accounting (slurmdbd) requires an external MariaDB and is disabled in defaults — see `recipes/components/slinky-slurm/values.yaml`. | [Slinky Slurm Cluster Chart](https://github.com/SlinkyProject/slurm-operator/tree/main/helm/slurm) |
+| **slinky-topograph** | Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm `topology.conf`, enabling topology-aware placement decisions in the Slinky-managed scheduler. **Not installed by default**; leaf overlays opt in by adding an explicit `componentRef` entry for `slinky-topograph` — the `componentRef` is what schedules the release; `dependencyRefs` alone does not install anything. That `componentRef` declares `slinky-slurm` as a `dependencyRef` to deploy **after** it: `slinky-slurm` renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first. `TopologyPlugin: topology/tree` is set per-leaf via `slinky-slurm`'s `controller.extraConfMap`. Includes the `node-observer` sub-chart, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP `roles/compute.viewer` for Workload Identity). | [Topograph](https://github.com/NVIDIA/topograph) |
 | **nfd-ocp-olm** | OLM installer for Node Feature Discovery on OpenShift. Creates the OperatorGroup and Subscription resources that install NFD via the Operator Lifecycle Manager. Paired with `nfd-ocp`. OCP-specific. | [Node Feature Discovery (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5ec53e8c110f56bd24f5f8db) |
 | **nfd-ocp** | Node Feature Discovery CR for OpenShift. Configures NFD's operand (worker, topology updater) via a NodeFeatureDiscovery custom resource. Deployed after `nfd-ocp-olm`. OCP-specific. | [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) |
 | **gpu-operator-ocp-olm** | OLM installer for the GPU Operator on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified GPU Operator via the Operator Lifecycle Manager. Paired with `gpu-operator-ocp`. OCP-specific. | [NVIDIA GPU Operator (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5e7b210b8a3c1e00013d636d) |
@@ -54,6 +55,7 @@ Not every component appears in every recipe. The recipe engine selects component
 - **Cloud-specific components** (aws-efa, aws-ebs-csi-driver) are added when the service matches. OCP recipes replace base components (gpu-operator, nfd, network-operator) with OLM+CR pairs (e.g., `gpu-operator-ocp-olm` + `gpu-operator-ocp`).
 - **Intent-specific components** (agentgateway, agentgateway-crds) are added based on workload intent (e.g., inference recipes include the inference gateway).
 - **Platform-specific components** (slinky-slurm-operator, slinky-slurm, kubeflow-trainer, dynamo-platform) are added when the recipe selects a matching `--platform`. For `--platform slurm`, all three core Slinky pieces (`slinky-slurm-operator-crds`, `slinky-slurm-operator`, `slinky-slurm`) are declared inline per slurm leaf overlay — the same shape `dynamo-platform` uses across `*-inference-dynamo` leaves. IMEX-capable Slurm leaves attach a fixed ComputeDomain through `slinky-slurm.preManifestFiles` so slurmd pods can consume DRA-provisioned IMEX channels. Leaves that want the operator only inline the CRDs + operator and omit the `slinky-slurm` componentRef. For an end-to-end walkthrough (recipe → bundle → install → validate → `srun` smoke job on EKS, GKE, or Kind), see [`demos/cuj1-slinky-slurm.md`](https://github.com/NVIDIA/aicr/blob/main/demos/cuj1-slinky-slurm.md).
+- **Topology-aware optional components** (`slinky-topograph`) are not installed by default. Opting in requires an explicit `componentRef` entry for `slinky-topograph` in the leaf overlay — the `componentRef` is what installs it; `dependencyRefs` alone does not. That `componentRef` declares `slinky-slurm` as a `dependencyRef`, so Topograph deploys after the Slurm cluster chart, which owns the ConfigMap Topograph patches. See the wiring example in the [Recipe Development Guide](../integrator/recipe-development.md#slinky-slurm-inline-components).
 - **Accelerator/OS-specific tuning** (nodewright-customizations, nvidia-dra-driver-gpu) varies by hardware and OS combination.
 
 ### NFD Topology Updater

docs/user/container-images.md

@@ -19,8 +19,8 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by
 <!-- BEGIN AICR-BOM -->
 ## Summary
 
-- Components: **33**
-- Unique images: **85**
+- Components: **34**
+- Unique images: **86**
 - Distinct registries: **11**
 
 Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `public.ecr.aws`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev`
@@ -64,6 +64,7 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip
 | slinky-slurm | helm | slurm | 1.2.0 | 5 |
 | slinky-slurm-operator | helm | slurm-operator | 1.2.0 | 2 |
 | slinky-slurm-operator-crds | helm | slurm-operator-crds | 1.2.0 | 0 |
+| slinky-topograph | helm | topograph/topograph | 0.5.0 | 1 |
 
 ## Version variants
 
@@ -265,6 +266,10 @@ _No images extracted._
 
 _No images extracted._
 
+### slinky-topograph
+
+- `ghcr.io/nvidia/topograph:v0.5.0`
+
 ### kube-prometheus-stack@83.7.0 (variant)
 
 - `docker.io/grafana/grafana:12.4.3`

pkg/recipe/deployment_order_guard_test.go

@@ -236,12 +236,20 @@ func TestDeploymentOrderGuards(t *testing.T) {
 			requiredDeps: map[string][]string{
 				"slinky-slurm-operator": {"cert-manager", "slinky-slurm-operator-crds"},
 				"slinky-slurm":          {"slinky-slurm-operator", "slinky-slurm-operator-crds"},
+				"slinky-topograph":      {"slinky-slurm-operator", "slinky-slurm-operator-crds", "slinky-slurm"},
 			},
 			requiredOrdering: [][2]string{
 				{"cert-manager", "slinky-slurm-operator"},
 				{"slinky-slurm-operator-crds", "slinky-slurm-operator"},
 				{"slinky-slurm-operator", "slinky-slurm"},
 				{"slinky-slurm-operator-crds", "slinky-slurm"},
+				// slinky-topograph deploys AFTER slinky-slurm: topograph's slinky
+				// engine create-or-updates the ConfigMap named by
+				// topologyConfigmapName (slinky-slurm-config-extra). The slurm
+				// chart must create that CM first or Helm install fails on
+				// ownership metadata. Topograph then patches only the
+				// topology.conf key (maps.Copy), preserving chart-owned keys.
+				{"slinky-slurm", "slinky-topograph"},
 				{"gpu-operator", "nvsentinel"},
 			},
 		},
@@ -258,12 +266,20 @@ func TestDeploymentOrderGuards(t *testing.T) {
 			requiredDeps: map[string][]string{
 				"slinky-slurm-operator": {"cert-manager", "slinky-slurm-operator-crds"},
 				"slinky-slurm":          {"slinky-slurm-operator", "slinky-slurm-operator-crds"},
+				"slinky-topograph":      {"slinky-slurm-operator", "slinky-slurm-operator-crds", "slinky-slurm"},
 			},
 			requiredOrdering: [][2]string{
 				{"cert-manager", "slinky-slurm-operator"},
 				{"slinky-slurm-operator-crds", "slinky-slurm-operator"},
 				{"slinky-slurm-operator", "slinky-slurm"},
 				{"slinky-slurm-operator-crds", "slinky-slurm"},
+				// slinky-topograph deploys AFTER slinky-slurm: topograph's slinky
+				// engine create-or-updates the ConfigMap named by
+				// topologyConfigmapName (slinky-slurm-config-extra). The slurm
+				// chart must create that CM first or Helm install fails on
+				// ownership metadata. Topograph then patches only the
+				// topology.conf key (maps.Copy), preserving chart-owned keys.
+				{"slinky-slurm", "slinky-topograph"},
 			},
 		},
 	}