NemoClaw Kubernetes GPU autoscaling

Experimental community recipe: a CPU-only NemoClaw/OpenClaw sandbox (OpenShell) sends inference to authenticated Ollama pods in the same cluster. An HPA scales only those Ollama pods. This recipe’s example signal is per-pod GPU utilization; you can switch the HPA to other or custom metrics. Unsupported / non-production.

Envoy Gateway is optional. Use it for LeastRequest across GPU replicas; skip it when the agent ClusterIP Service is enough:

Choice	Install command
With Envoy LeastRequest (default)	`./scripts/install-hpa.sh`
Without Envoy (agent Service only)	`ENABLE_ENVOY_LB=0 ./scripts/install-hpa.sh`

New here? Start with Quick start. Teardown: Uninstall.

Pins in versions.env: NemoClaw v0.0.104, OpenShell 0.0.85, Agent Sandbox v0.5.0.

Architecture

OpenShell CLI → port-forward → OpenShell gateway → CPU-only NemoClaw sandbox

Runtime inference path (HPA scales to N Ollama pods, 1 GPU each). Envoy is optional: LeastRequest when enabled; agent ClusterIP Service when ENABLE_ENVOY_LB=0. Set MAX_REPLICAS / TARGET_PODS from allocatable GPUs — not fixed to 4.

OpenShell CPU sandbox
        ↓
Envoy Gateway — LeastRequest  (or agent Service when ENABLE_ENVOY_LB=0)
        ↓
Authenticated inference endpoints
├─ Ollama pod → GPU 1
├─ Ollama pod → GPU 2
├─ …
└─ Ollama pod → GPU N
        ↑
HPA (example: GPU utilization)

Inference API key. Chart-generated local Secret for Bearer auth on /v1/models and chat completions; users do not supply a cloud key. OpenShell injects it for the sandbox — not for Ollama model pulls, and not OpenAI/NVIDIA_API_KEY.

HPA metrics (example). The shipped path uses GPU utilization:

DCGM_FI_DEV_GPU_UTIL → Prometheus → Adapter gpu_utilization_percent → HPA

That is only an example. Point the HPA at other Prometheus Adapter custom metrics (or your own) by changing monitoring/prometheus-adapter-gpu-values.yaml and the chart autoscaling settings — for example request rate, queue depth, or another DCGM signal.

Default	Value
Namespace / release	`nemoclaw-gpu`
Service port	`8081`
Model	`llama3.2:3b`
GPUs per pod / min–max replicas	`1` / `1`–`4`
HPA GPU target (example metric)	`40%`
Ingress host (example)	`nemoclaw.local`

Boundaries (short): namespaces nemoclaw-gpu and nemoclaw-sandboxes; only Ollama pods request GPUs; Envoy dataplane must stay ClusterIP while the OpenShell cleartext HTTP listener exists (NodePort/LoadBalancer rejected); chart creates no NetworkPolicy; installer may touch shared Prometheus, Adapter, Envoy, DCGM ServiceMonitor, MicroK8s add-ons. Validated on one MicroK8s node with 4× L40S — re-validate other hardware.

Prerequisites

Kubernetes 1.29+, Helm 3, kubectl
Allocatable nvidia.com/gpu, nodes labeled nvidia.com/gpu.present=true
NVIDIA GPU Operator with DCGM Exporter; Metrics Server
Docker Buildx + a registry every node can pull
OpenShell CLI matching versions.env (for example uv tool install "openshell==${OPENSHELL_VERSION}")
OIDC issuer/client for the default OpenShell path (or the documented unauthenticated eval exception)

Use a dedicated evaluation cluster. Do not paste kubeconfigs, registry creds, OIDC secrets, or inference API keys into issues/PRs.

Quick start

From an empty clone to a working sandbox. Run from examples/recipes/nvidia/kubernetes-gpu-autoscaling/ unless noted. Deeper options: Install details, OpenShell details.

1. Clone and tools

git clone https://github.com/NVIDIA/nemoclaw-community.git
cd nemoclaw-community/examples/recipes/nvidia/kubernetes-gpu-autoscaling
source versions.env
uv tool install "openshell==${OPENSHELL_VERSION}"
openshell --version

2. Confirm GPUs and DCGM

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}{" GPUs="}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'
kubectl get pods -n gpu-operator-resources -l app=nvidia-dcgm-exporter

3. Install GPU inference + HPA

TLS is required by default when Envoy is on (see Install details). Isolated eval only: ALLOW_INSECURE_HTTP=1.

export HPA_VALUES=/path/to/hpa-tls-values.yaml
export INGRESS_HOST=nemoclaw.example.com
# Optional: export NEMOCLAW_TARGET_NODE=<gpu-node-name>
MAX_REPLICAS=2 ./scripts/install-hpa.sh
# Or without Envoy: ENABLE_ENVOY_LB=0 MAX_REPLICAS=2 ./scripts/install-hpa.sh

Wait for the first Ollama model pull (ROLLOUT_TIMEOUT if needed). Then:

kubectl get pods,service,hpa -n nemoclaw-gpu
./scripts/get-hpa.sh -n nemoclaw-gpu

Optional smoke test: Call inference.

4. Agent Sandbox, image, OpenShell

source versions.env
kubectl apply -f \
  "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/manifest.yaml"

export NEMOCLAW_SANDBOX_IMAGE=registry.example.com/team/nemoclaw-k8s:v0.0.104
./scripts/build-nemoclaw-sandbox-image.sh

export OPENSHELL_OIDC_ISSUER=https://idp.example.com/realms/openshell
export OPENSHELL_OIDC_AUDIENCE=openshell-cli
./scripts/install-openshell-k8s.sh

Dedicated eval without OIDC: ALLOW_UNAUTHENTICATED_OPENSHELL=1 plus OPENSHELL_UNAUTHENTICATED_ACK=dedicated-cluster-port-forward-only.

5. Connect CLI and create sandbox

Terminal 1 — keep running:

kubectl -n nemoclaw-sandboxes port-forward service/openshell 8080:8080

Terminal 2 — client TLS + gateway (OIDC flags in OpenShell details), then:

export NEMOCLAW_SANDBOX_IMAGE=registry.example.com/team/nemoclaw-k8s:v0.0.104
export INFERENCE_MODEL=llama3.2:3b
./scripts/create-nemoclaw-sandbox.sh
./scripts/verify-nemoclaw-sandbox.sh
./scripts/run-nemoclaw-sandbox.sh   # keep in foreground

Users do not paste an inference API key; the chart generates it and OpenShell injects Bearer auth.

6. Optional HPA / Envoy check

./scripts/hpa-load-test.sh

When finished: Uninstall.

Install details

Installer side effects: may install/upgrade Prometheus (if missing), Prometheus Adapter (always, with this recipe’s GPU rules), Envoy Gateway (when ENABLE_ENVOY_LB=1), DCGM ServiceMonitor, and MicroK8s GPU/Metrics add-ons. Review shared-cluster impact before reuse of release names.

Static checks (no cluster):

./scripts/test-render-contract.sh
./scripts/test-script-security-contract.sh
node ./scripts/test-inference-auth-contract.mjs
./scripts/test-nemoclaw-k8s-contract.sh

TLS values

kubectl create namespace nemoclaw-gpu --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret tls nemoclaw-example-tls \
  --namespace nemoclaw-gpu \
  --cert=/path/to/tls.crt --key=/path/to/tls.key \
  --dry-run=client -o yaml | kubectl apply -f -
cp values-step2-hpa.yaml /path/to/hpa-tls-values.yaml

# in hpa-tls-values.yaml
ingress:
  host: nemoclaw.example.com
  tls:
    - secretName: nemoclaw-example-tls
      hosts:
        - nemoclaw.example.com

export HPA_VALUES=/path/to/hpa-tls-values.yaml
export INGRESS_HOST=nemoclaw.example.com

The chart does not create, rotate, or delete the TLS Secret.

Scheduling

Unset NEMOCLAW_TARGET_NODE for portable scheduling. Multi-node needs RWX (or disable Ollama persistence); values-step2-hpa.yaml hostPath is single-node only.
Pin with export NEMOCLAW_TARGET_NODE=<exact-node-name> after confirming Ready + GPU label + allocatable GPUs ≥ MAX_REPLICAS.
MAX_REPLICAS / TARGET_PODS must not exceed allocatable GPUs in scope. Host nvidia-smi processes outside Kubernetes are not reserved by the chart.
Keep HPA_VALUES, INGRESS_HOST, ENABLE_ENVOY_LB, and NEMOCLAW_TARGET_NODE consistent across install-hpa.sh, hpa-reset.sh, and hpa-load-test.sh.

Ingress security

When Envoy is enabled:

Dataplane Service type is ClusterIP only. NodePort / LoadBalancer are rejected so the hostname-unrestricted OpenShell cleartext HTTP listener is not exposed externally. Use kubectl port-forward from outside the cluster.
External HTTPS route: Gateway Basic auth + inference key as X-Api-Key (Basic owns Authorization).
OpenShell HTTPRoute: no Gateway Basic auth so OpenShell can inject Authorization: Bearer.
TLS required by default. Isolated eval cleartext: ALLOW_INSECURE_HTTP=1 (ClusterIP only). Preflight checks Kubernetes-reported exposure; it does not prove private-network isolation. Set per script invocation.
Auth Secrets (nemoclaw-gpu-agent-inference-api, nemoclaw-gpu-agent-ingress-auth) use Helm keep. Delete explicitly to rotate; never commit keys. Optional operator Secret: inference.auth.existingSecret.

When Envoy is disabled (ENABLE_ENVOY_LB=0): no Gateway objects; clients use the agent Service; protect with network policy and the inference API key.

Ollama storage

Default persistence off (emptyDir per pod → re-pull on replace). Single-node hostPath: values-step2-hpa.yaml. Multi-node: RWX StorageClass + persistence.enabled=true.

Recovery

Destructive recovery for the selected release only: ./scripts/cluster-recover.sh (optional RESTART_MICROK8S=1). See script comments before use.

Verify

kubectl get pods,service,hpa -n nemoclaw-gpu
kubectl get --raw \
  '/apis/custom.metrics.k8s.io/v1beta1/namespaces/nemoclaw-gpu/pods/*/gpu_utilization_percent'
./scripts/get-hpa.sh -n nemoclaw-gpu
./scripts/get-agent-pods.sh -n nemoclaw-gpu

Idle expectation: one Running inference pod (two containers), HPA at one replica targeting current/40.

Call inference

Operator port-forward bypasses Gateway TLS/Basic; Bearer still required. Do not bind to a non-loopback address.

kubectl port-forward -n nemoclaw-gpu service/nemoclaw-gpu-agent 8081:8081

curl -s http://127.0.0.1:8081/healthz
INFERENCE_API_KEY="$(kubectl get secret nemoclaw-gpu-agent-inference-api \
  -n nemoclaw-gpu -o jsonpath='{.data.api-key}' | base64 -d)"
curl -s http://127.0.0.1:8081/v1/models \
  -H "Authorization: Bearer ${INFERENCE_API_KEY}"
unset INFERENCE_API_KEY

/healthz, /readyz, /metrics are unauthenticated. /readyz may be 503 during the initial model download.

OpenShell details

Agent Sandbox CRDs are cluster-scoped; install-openshell-k8s.sh never installs them — apply the pinned manifest yourself.
Build image: NEMOCLAW_SANDBOX_IMAGE=… ./scripts/build-nemoclaw-sandbox-image.sh (versioned, non-latest tag; no API key in the image).
OIDC is default. Unauthenticated mode is dedicated-cluster + port-forward only (ALLOW_UNAUTHENTICATED_OPENSHELL=1 + ACK). ClusterIP does not isolate from other pods/users.
Client mTLS after port-forward:

MTLS_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/openshell/gateways/nemoclaw-k8s/mtls"
mkdir -p "${MTLS_DIR}"
for key in ca.crt tls.crt tls.key; do
  kubectl get secret openshell-client-tls -n nemoclaw-sandboxes \
    -o "jsonpath={.data.${key//./\\.}}" | base64 -d >"${MTLS_DIR}/${key}"
done
chmod 600 "${MTLS_DIR}"/*
openshell gateway add https://127.0.0.1:8080 \
  --local --name nemoclaw-k8s \
  --oidc-issuer "${OPENSHELL_OIDC_ISSUER}" \
  --oidc-client-id "${OPENSHELL_OIDC_CLIENT_ID:-openshell-cli}" \
  --oidc-audience "${OPENSHELL_OIDC_AUDIENCE}"
# Unauth eval: omit --oidc-* flags
openshell status

create-nemoclaw-sandbox.sh stores the chart inference key in the OpenShell provider, strips integrate.api.nvidia.com from policy, and verifies models + a short chat completion.
OpenShell 0.0.85 leaves sandboxes idle (sleep infinity); run-nemoclaw-sandbox.sh must stay attached and does not auto-restart. Combined topology may require powerful capabilities (SYS_ADMIN, NET_ADMIN, …) — check admission policy.

Test autoscaling and load balancing

hpa-load-test.sh (when SCALE_UP_TARGET ≥ 2):

Scale-up/down via direct pod-IP load (bypasses Envoy on purpose).
When Envoy is on: concurrent Bearer traffic through Envoy; every Ready pod must get successes and no pod may exceed LB_TEST_MAX_SHARE (default 0.75).

HPA_VALUES=/path/to/hpa-tls-values.yaml INGRESS_HOST=nemoclaw.example.com \
TARGET_PODS=2 SCALE_UP_TARGET=2 ./scripts/hpa-load-test.sh
# then optionally TARGET_PODS=4 SCALE_UP_TARGET=4
./scripts/hpa-reset.sh

Knob	Default	Purpose
`SKIP_ENVOY_LB_TEST`	`0`	Skip Envoy distribution phase
`ENABLE_ENVOY_LB`	`1`	Keep consistent with install
`LB_TEST_REQUESTS` / `LB_TEST_CONCURRENCY`	`48` / `12`	Envoy check load
`TARGET_PODS` / `SCALE_UP_TARGET`	allocatable GPUs	HPA test ceiling
`DURATION_SEC` / `HPA_TARGET_GPU`	`720` / `40`	Load duration / util target

Grafana is optional visualization; the script is the pass/fail check.

Scripts

Script	Purpose
`install-hpa.sh`	Monitoring + chart + HPA (+ Envoy if enabled)
`hpa-load-test.sh` / `hpa-reset.sh`	Autoscaling (+ Envoy) test / restore idle
`cluster-recover.sh`	Destructive release recovery
`get-agent-pods.sh` / `get-hpa.sh` / `hpa-watch.sh`	Inspect / watch
`build-nemoclaw-sandbox-image.sh`	Build/push sandbox image
`install-openshell-k8s.sh`	OpenShell gateway
`create-nemoclaw-sandbox.sh` / `verify-nemoclaw-sandbox.sh` / `run-nemoclaw-sandbox.sh`	Sandbox lifecycle
`test--contract.`	Static / local contract checks

Grafana

Optional. After install-hpa.sh:

kubectl port-forward -n monitoring service/kube-prometheus-grafana 3000:80
# http://127.0.0.1:3000 — creds from secret kube-prometheus-grafana in monitoring

Explore → Prometheus:

avg by (exported_pod) (
  DCGM_FI_DEV_GPU_UTIL{exported_namespace="nemoclaw-gpu", exported_pod=~"nemoclaw-gpu-agent-.*"}
)

sum by (pod) (
  rate(nemoclaw_llm_requests_total{namespace="nemoclaw-gpu", result="success"}[5m])
)

Uninstall

Stop run-nemoclaw-sandbox.sh. With OpenShell port-forward still up:

openshell sandbox delete nemoclaw-onprem
openshell provider delete onprem-ollama
openshell gateway remove nemoclaw-k8s
rm -r -- "${XDG_CONFIG_HOME:-${HOME}/.config}/openshell/gateways/nemoclaw-k8s/mtls"

helm uninstall openshell -n nemoclaw-sandboxes
helm uninstall nemoclaw-gpu -n nemoclaw-gpu

Optional explicit cleanup (only if this recipe owns them): OpenShell PVC/Secrets in nemoclaw-sandboxes; nemoclaw-gpu-agent-inference-api and nemoclaw-gpu-agent-ingress-auth; dedicated namespaces after inspection. Agent Sandbox CRDs are cluster-scoped — delete the pinned manifest only if no other sandboxes remain.

Does not remove shared Prometheus, Adapter, Envoy Gateway, DCGM ServiceMonitor, or MicroK8s add-ons. Review ownership before removing those.

Known limitations

Experimental, unsupported; OpenShell Kubernetes driver is experimental.
Idle sandbox + foreground run-nemoclaw-sandbox.sh; no auto-restart; privilege separation runs as sandbox identity.
No NetworkPolicy; Bearer auth is not network isolation.
Example HPA signal is GPU utilization; other/custom metrics need Adapter + HPA edits. hostPath model cache is single-node.
First Ollama start and sandbox image pulls depend on registry access.
Installer mutates shared cluster components — review with the cluster admin.

Third-party notices: THIRD-PARTY-NOTICES. Update pins in versions.env as one compatibility contract, then rebuild the sandbox image and recreate the sandbox.

Navigation Menu

Uh oh!

NemoClaw Kubernetes GPU autoscaling

Architecture

Prerequisites

Quick start

1. Clone and tools

2. Confirm GPUs and DCGM

3. Install GPU inference + HPA

4. Agent Sandbox, image, OpenShell

5. Connect CLI and create sandbox

6. Optional HPA / Envoy check

Install details

TLS values

Scheduling

Ingress security

Ollama storage

Recovery

Verify

Call inference

OpenShell details

Test autoscaling and load balancing

Scripts

Grafana

Uninstall

Known limitations

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

NemoClaw Kubernetes GPU autoscaling

Architecture

Prerequisites

Quick start

1. Clone and tools

2. Confirm GPUs and DCGM

3. Install GPU inference + HPA

4. Agent Sandbox, image, OpenShell

5. Connect CLI and create sandbox

6. Optional HPA / Envoy check

Install details

TLS values

Scheduling

Ingress security

Ollama storage

Recovery

Verify

Call inference

OpenShell details

Test autoscaling and load balancing

Scripts

Grafana

Uninstall

Known limitations