feat(examples): add Kubernetes GPU autoscaling recipe

[maggiezha]

Related Issue

N/A. This moves the public work from NVIDIA/NemoClaw pull request #7459 into the repository designated for community examples, and updates the load balancer from ingress-nginx to Envoy Gateway with LeastRequest.

Description

Add an experimental NVIDIA-authored Kubernetes recipe that runs a CPU-only NemoClaw/OpenClaw agent in an OpenShell sandbox and sends inference through Envoy Gateway (LeastRequest) to authenticated Ollama replicas on NVIDIA GPUs in the same cluster. A Kubernetes Horizontal Pod Autoscaler scales only the Ollama inference pods from the per-pod gpu_utilization_percent metric through DCGM Exporter, Prometheus, and Prometheus Adapter.

The runtime path is entirely on-premises:

OpenShell CPU sandbox
        ↓
Envoy Gateway — LeastRequest
        ↓
Authenticated inference endpoints
├─ Ollama pod → GPU 1
├─ Ollama pod → GPU 2
├─ Ollama pod → GPU 3
└─ Ollama pod → GPU 4
        ↑
GPU-utilization HPA

The sandbox image contains no inference credential. OpenShell holds the in-cluster API key and injects it as Bearer when forwarding requests through the Envoy dataplane to the inference pods. The setup does not use NVIDIA_API_KEY or a hosted GPU endpoint, and the sandbox policy removes and verifies removal of the inherited integrate.api.nvidia.com endpoint before the agent starts.

Source migration

Source	Destination
NVIDIA/NemoClaw: deploy/helm/gpu_autoscaling_k8s/	NVIDIA/nemoclaw-community: examples/recipes/nvidia/kubernetes-gpu-autoscaling/

• Imported chart files from exact source head 77334ccbbadba2c5079fe9e99fe80a3cddc846b5.
• Preserve compatibility-sensitive Helm release, namespace, Service, metric, and label identifiers.
• Make the community directory the canonical source after migration. Future compatibility changes should be reviewed and applied here rather than mirrored from the old pull-request branch.
• Pin the experimental integration to NemoClaw v0.0.104 at f389c9d872775006ae069473f58250fa8f3ad40f, OpenShell 0.0.85, and Agent Sandbox v0.5.0.
• Follow-up on this branch replaces ingress-nginx with Envoy Gateway and pins LeastRequest on the OpenShell and external inference routes.

NemoClaw and OpenShell path

• Build and push a deployment-specific NemoClaw image without embedding credentials.
• Install an internal-only OpenShell Kubernetes gateway after the operator explicitly installs the pinned Agent Sandbox controller.
• Require OIDC by default. The unauthenticated mode requires a separate dedicated-cluster, port-forward-only acknowledgement.
• Configure the OpenShell inference provider to the in-cluster Envoy Gateway dataplane (LeastRequest), which forwards Bearer-authenticated requests to the Ollama inference pods.
• Create the NemoClaw sandbox without a GPU request, remove the hosted-inference endpoint, and run NemoClaw in a foreground OpenShell exec session.
• Document the OpenShell 0.0.85 idle-sandbox lifecycle, capabilities, privilege-separation limitation, cleanup, and coordinated pin-update procedure.

Security and lifecycle boundaries

• Require a timing-safe inference API key check for model and completion traffic on the in-cluster proxy (Authorization: Bearer on the OpenShell/Envoy path, or X-Api-Key for external clients when Gateway Basic auth already owns Authorization). Health, readiness, and metrics remain unauthenticated.
• Generate and retain a random inference API key, or preserve an operator-managed Secret name and key end to end. Operational scripts discover the effective Secret contract from the installed Helm release.
• Keep TLS required by default. Enable Gateway Basic authentication on the external HTTPS HTTPRoute; the OpenShell HTTPRoute does not use Gateway Basic auth so Bearer can pass through Envoy.
• State that direct Service access bypasses Gateway TLS and Basic authentication but not the application API-key check, and that the chart creates no NetworkPolicy.
• Document Prometheus Adapter, Envoy Gateway, DCGM ServiceMonitor, Agent Sandbox, OpenShell, and MicroK8s mutations and retained resources.
• Prefer staged validation: start with two replicas, then raise TARGET_PODS / MAX_REPLICAS to the node's allocatable GPU count.
• Synchronize the HPA replica maximum and GPU safety ceiling so explicit staged targets are not capped by an outdated lower maxGpus default.
• Make Helm uninstall the default cleanup. Namespace deletion requires an explicit exclusive-ownership check.

This remains an unsupported, non-production community experiment; it does not establish native Kubernetes as a supported NemoClaw product surface.

Verification

• python3 scripts/check_license_headers.py --check
• python3 scripts/check_label_taxonomy.py
• python3 -m unittest scripts.tests.test_governance_taxonomy
• python3 scripts/check_pr_title.py --advisory 'feat(examples): add Kubernetes GPU autoscaling recipe'
• git diff --check
• Bash syntax validation for the recipe scripts
• bash scripts/test-script-security-contract.sh
• bash scripts/test-nemoclaw-k8s-contract.sh
• node scripts/test-inference-auth-contract.mjs
• bash scripts/test-render-contract.sh with Helm 3.20.2 on PATH
• helm lint . --set ingress.allowInsecureHttp=true
• Render coverage for mandatory inference authentication, TLS redirect, operator-managed Secrets, long Secret names, scalar-like and dotted Secret keys, RWX persistence, and single-node hostPath
• Current branch is based on the latest fetched upstream/main; it is zero commits behind
• Current head 196fba6 is signed and reported Verified by GitHub
• Live Kubernetes/GPU validation on 4× NVIDIA L40S (MicroK8s): chart deploy, Envoy Gateway LeastRequest, authenticated inference (external Basic + X-Api-Key; OpenShell-path Bearer through Envoy), gpu_utilization_percent HPA signal, scale-up/scale-down via hpa-load-test.sh, and Envoy distribution check (concurrent chat completions across Ready GPU pods). Full OpenShell/NemoClaw sandbox-image creation on this node remains optional follow-up and is not claimed as live-validated here.

Documentation Writer Review

• Documentation writer review completed for the final changes
• Result: docs-updated
• Evidence or justification: Independent review covered the README, catalog, notices, Helm notes, scripts, templates, and tests. It confirmed the pinned lifecycle and capability claims, CPU-only sandbox boundary, authenticated on-premises Envoy route, hosted-endpoint removal, operator-managed Secret behavior, retained-resource cleanup, synchronized GPU ceiling, and L40S-oriented staged validation guidance. Bash syntax, Helm render, native Kubernetes, recovery/security, inference authentication, SPDX, and diff checks passed.
• Reviewer: Independent Codex documentation-writer review
• Reviewed head: 196fba6ed5434735752336c7d04a95dbbd301ca9
• Changed user-facing text follows the writing guide and controlled-word list.
• A public contributor can understand the changed text without internal company context.
• I reviewed any agent-generated text before submission. Contributor review is required before merge.

Release And Compliance

• No secrets or credentials are included, including API keys, access tokens, passwords, local .env files, private certificates, or token caches.
• No nonpublic project names, environment names, hostnames, URLs, ticket identifiers, workspace paths, logs, screenshots, or configuration values are included.
• Third-party dependency changes are reflected in THIRD-PARTY-NOTICES.
• Public content uses sanitized examples and placeholders instead of private values.
• I added my DCO sign-off declaration to this pull request description.

Signed-off-by: maggiezha maggiez@nvidia.com

[apurvvkumaria]

Thanks for moving this Kubernetes GPU autoscaling work into the community repository. The static checks and contract tests are in good shape, but two items need to be completed before approval:

1. Please provide a documented live Kubernetes/GPU validation of the core workflow. The recipe's primary behavior is GPU-backed inference and HPA scaling, but the PR currently states that no live Kubernetes/H100 validation was run. At minimum, validate deployment, authenticated inference, the custom GPU metric, and a scale-up/scale-down cycle on a suitable cluster.
2. Please complete the contributor review of the agent-generated text and check the corresponding PR-template item. The PR currently marks this as required before merge.

Once those are complete, we can reassess the PR for approval.

[maggiezha]

Thanks for moving this Kubernetes GPU autoscaling work into the community repository. The static checks and contract tests are in good shape, but two items need to be completed before approval:

1. Please provide a documented live Kubernetes/GPU validation of the core workflow. The recipe's primary behavior is GPU-backed inference and HPA scaling, but the PR currently states that no live Kubernetes/H100 validation was run. At minimum, validate deployment, authenticated inference, the custom GPU metric, and a scale-up/scale-down cycle on a suitable cluster.
2. Please complete the contributor review of the agent-generated text and check the corresponding PR-template item. The PR currently marks this as required before merge.

Once those are complete, we can reassess the PR for approval.

Thanks for the prompt reply: 1. I had to give up the DGX H100 test, because the admin already setup the cluster to have GPU time slicing to 4 replicas per GPU, which will affect my K8s pod and GPU utilization per GPU, this H100 test is pending on if the cluster admin agree to change GPU operator to cancel the GPU time slicing to exclusive GPU allocation. Instead, I had to go back to the same hardware I used for PR7459: Brev 4xL40S on AWS: NVIDIA L40S (44.70 GiB) 4 GPUs x 48 CPUs 384 GiB RAM

[apurvvkumaria]

Thank you for the update. The documented 4× L40S run now covers the chart deployment, authenticated inference, GPU metric, HPA scale-up/scale-down, and Envoy LeastRequest distribution. The contributor and documentation-review checklist items are also complete on the current head.

Two items still need attention before approval:

1. Separate the internal cleartext route from externally exposed Envoy Services. The current Gateway always creates a hostname-unrestricted HTTP listener and OpenShell route on port 80. When ingress.gateway.serviceType is changed to NodePort or LoadBalancer, that same listener becomes externally reachable even when TLS is configured. Requests using another Host value or the load-balancer IP can select the OpenShell route instead of the hostname-specific HTTPS redirect, bypassing Gateway TLS and Basic authentication while carrying the reusable Bearer credential and inference traffic over cleartext. Please either use a separate ClusterIP-only Gateway/dataplane for OpenShell and a distinct TLS-only external Gateway, or reject NodePort/LoadBalancer while the internal HTTP listener is present. Add a render/security contract test for the chosen boundary.
2. Complete or narrow the end-to-end validation claim. The L40S evidence validates the GPU chart and Envoy path, but the PR still advertises a NemoClaw/OpenShell sandbox workflow and explicitly states that sandbox creation was not live-validated. Please run the actual sandbox creation and inference path through https://inference.local, including a model request from the sandbox, or narrow the recipe's advertised scope so the unvalidated sandbox integration is not presented as a completed workflow.

The available local security, Kubernetes, inference-authentication, syntax, SPDX, taxonomy, and diff checks pass on 196fba6.