Llama-3.1-Nemotron-Nano-8B-v1 on OCI OKE (Private Deployment)

This cookbook documents a validated private deployment of nvidia/Llama-3.1-Nemotron-Nano-8B-v1 on Oracle Cloud Infrastructure (OCI) using a private OKE cluster, a single VM.GPU.A10.1 worker, and vLLM with an OpenAI-compatible /v1 endpoint.

Based on the Deploy OpenAI vLLM Production Stack on OKE guide, customized for the Nemotron model with tool calling support.

Tested environment

Region: us-phoenix-1
Kubernetes: OKE v1.31.10, enhanced cluster
GPU shape: VM.GPU.A10.1 (NVIDIA A10, 24 GB)
CPU shape: VM.Standard.E5.Flex
Model: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
Serving stack: vLLM v0.19.0
Helm chart: vllm/vllm-stack 0.1.10
Inference API: OpenAI-compatible /v1

Validated capabilities

Chat completion
Tool / function calling
Streaming
Async / concurrent requests
OpenAI-compatible model discovery via /v1/models

Prerequisites

OCI tenancy with GPU capacity (VM.GPU.A10.1)
oci CLI configured with a valid profile
kubectl, helm, ssh, jq
An SSH key pair (e.g., ~/.ssh/id_ed25519)

Note: The NVIDIA device plugin is pre-installed on OKE enhanced clusters. No manual installation is required.

Architecture

                          ┌─────────────────────────────────────────────────┐
                          │                  VCN 10.0.0.0/16               │
  You ──SSH tunnel──►     │                                                │
  (localhost:6443)        │  ┌──────────┐     ┌──────────────────────────┐  │
          │               │  │ Bastion  │     │   API subnet (private)   │  │
          │               │  │ subnet   │────►│   OKE control plane      │  │
          │               │  │ (public) │     │   :6443                  │  │
          ▼               │  └──────────┘     └──────────────────────────┘  │
  kubectl / curl          │                                                │
                          │  ┌──────────────────────────────────────────┐  │
                          │  │         Worker subnet (private)          │  │
                          │  │                                          │  │
                          │  │  ┌─────────────┐  ┌──────────────────┐  │  │
                          │  │  │ CPU node    │  │ GPU node (A10)   │  │  │
                          │  │  │ router pod  │  │ Nemotron engine  │  │  │
                          │  │  └─────────────┘  └──────────────────┘  │  │
                          │  └──────────────────────────────────────────┘  │
                          └─────────────────────────────────────────────────┘

Step 1: Set environment variables

export OCI_COMPARTMENT_ID="<your-compartment-ocid>"
export OCI_REGION="us-phoenix-1"
export OCI_PROFILE="DEFAULT"          # adjust to your OCI CLI profile
export CLUSTER_NAME="nemotron-phx"

# OKE retires older Kubernetes minors over time, so the example below may no
# longer be offered in your region. List the versions OKE currently supports
# and pick one of them:
#   oci ce cluster-options get --cluster-option-id all \
#       --compartment-id "${OCI_COMPARTMENT_ID}" \
#       --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
#       --query 'data."kubernetes-versions"'
export KUBERNETES_VERSION="v1.33.1"   # must be a version OKE currently lists

Step 2: Create VCN and networking

VCN_ID=$(oci network vcn create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --display-name "${CLUSTER_NAME}-vcn" \
    --cidr-blocks '["10.0.0.0/16"]' \
    --dns-label "nemotron" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

IGW_ID=$(oci network internet-gateway create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-igw" \
    --is-enabled true \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

NAT_ID=$(oci network nat-gateway create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-nat" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

SGW_SERVICE_ID=$(oci network service list \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data[?contains(name, 'All') && contains(name, 'Services')].id | [0]" \
    --raw-output)

SGW_SERVICE_NAME=$(oci network service list \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data[?contains(name, 'All') && contains(name, 'Services')].\"cidr-block\" | [0]" \
    --raw-output)

SGW_ID=$(oci network service-gateway create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-sgw" \
    --services "[{\"serviceId\": \"${SGW_SERVICE_ID}\"}]" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

PRIVATE_RT_ID=$(oci network route-table create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-private-rt" \
    --route-rules "[
        {\"cidrBlock\": \"0.0.0.0/0\", \"networkEntityId\": \"${NAT_ID}\"},
        {\"destination\": \"${SGW_SERVICE_NAME}\", \"destinationType\": \"SERVICE_CIDR_BLOCK\", \"networkEntityId\": \"${SGW_ID}\"}
    ]" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

PUBLIC_RT_ID=$(oci network route-table create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-public-rt" \
    --route-rules "[{\"cidrBlock\": \"0.0.0.0/0\", \"networkEntityId\": \"${IGW_ID}\"}]" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

SL_ID=$(oci network security-list create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-sl" \
    --egress-security-rules '[{"destination": "0.0.0.0/0", "protocol": "all", "isStateless": false}]' \
    --ingress-security-rules '[
        {"source": "0.0.0.0/0", "protocol": "6", "isStateless": false, "tcpOptions": {"destinationPortRange": {"min": 22, "max": 22}}},
        {"source": "10.0.0.0/16", "protocol": "all", "isStateless": false},
        {"source": "10.244.0.0/16", "protocol": "all", "isStateless": false},
        {"source": "10.96.0.0/16", "protocol": "all", "isStateless": false},
        {"source": "0.0.0.0/0", "protocol": "1", "isStateless": false, "icmpOptions": {"type": 3, "code": 4}}
    ]' \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

Create four subnets:

API_SUBNET_ID=$(oci network subnet create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-api-subnet" \
    --cidr-block "10.0.0.0/28" \
    --route-table-id "${PRIVATE_RT_ID}" \
    --security-list-ids "[\"${SL_ID}\"]" \
    --dns-label "kubeapi" \
    --prohibit-public-ip-on-vnic true \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

WORKER_SUBNET_ID=$(oci network subnet create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-worker-subnet" \
    --cidr-block "10.0.10.0/24" \
    --route-table-id "${PRIVATE_RT_ID}" \
    --security-list-ids "[\"${SL_ID}\"]" \
    --dns-label "workers" \
    --prohibit-public-ip-on-vnic true \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

LB_SUBNET_ID=$(oci network subnet create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-lb-subnet" \
    --cidr-block "10.0.20.0/24" \
    --route-table-id "${PUBLIC_RT_ID}" \
    --security-list-ids "[\"${SL_ID}\"]" \
    --dns-label "loadbalancers" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

BASTION_SUBNET_ID=$(oci network subnet create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --vcn-id "${VCN_ID}" \
    --display-name "${CLUSTER_NAME}-bastion-subnet" \
    --cidr-block "10.0.30.0/24" \
    --route-table-id "${PUBLIC_RT_ID}" \
    --security-list-ids "[\"${SL_ID}\"]" \
    --dns-label "bastion" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

Step 3: Create private OKE cluster

oci ce cluster create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --name "${CLUSTER_NAME}" \
    --vcn-id "${VCN_ID}" \
    --kubernetes-version "${KUBERNETES_VERSION}" \
    --endpoint-subnet-id "${API_SUBNET_ID}" \
    --service-lb-subnet-ids "[\"${LB_SUBNET_ID}\"]" \
    --endpoint-public-ip-enabled false \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

Wait for the cluster to become ACTIVE (~10 minutes):

# Poll until ACTIVE
CLUSTER_ID=$(oci ce cluster list \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --name "${CLUSTER_NAME}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data[0].id' --raw-output)

watch -n 30 "oci ce cluster get --cluster-id ${CLUSTER_ID} \
    --profile ${OCI_PROFILE} --region ${OCI_REGION} \
    --query 'data.\"lifecycle-state\"' --raw-output"

Do not proceed to Step 5 until the cluster is ACTIVE.

Step 4: Create OCI Bastion

The bastion is placed on the public bastion subnet so the OCI Bastion managed service can accept inbound SSH connections. The port-forwarding session then tunnels traffic to the private API endpoint over VCN-internal routing.

Known issue on OpenSSH 10.x (macOS 15+, some recent Linux): port-forwarding sessions close immediately after auth. If ssh -V reports 10.x, use Appendix A instead.

BASTION_ID=$(oci bastion bastion create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --bastion-type STANDARD \
    --target-subnet-id "${BASTION_SUBNET_ID}" \
    --name "${CLUSTER_NAME}-bastion" \
    --client-cidr-list "[\"$(curl -s https://ifconfig.me)/32\"]" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

Step 5: Create node pools

Find the GPU-compatible node image and create both pools. The OKE-<ver>- match is anchored with a trailing dash on purpose: a bare OKE-1.33.1 also substring-matches OKE-1.33.10 images, which would pick a node image newer than the control plane and be rejected for version skew.

GPU_IMAGE_ID=$(oci ce node-pool-options get \
    --node-pool-option-id all \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.sources[?contains(\"source-name\", 'GPU') && \
             contains(\"source-name\", 'OKE-${KUBERNETES_VERSION#v}-')].\"image-id\" | [0]" \
    --raw-output)

CPU_IMAGE_ID=$(oci ce node-pool-options get \
    --node-pool-option-id all \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.sources[?contains(\"source-name\", 'OKE-${KUBERNETES_VERSION#v}-') && \
             !contains(\"source-name\", 'GPU') && \
             contains(\"source-name\", 'aarch64')==\`false\`].\"image-id\" | [0]" \
    --raw-output)

# Verify both image IDs were found
echo "GPU image: ${GPU_IMAGE_ID}"
echo "CPU image: ${CPU_IMAGE_ID}"
# If either is empty, list available images and pick manually:
# oci ce node-pool-options get --node-pool-option-id all \
#     --compartment-id "${OCI_COMPARTMENT_ID}" \
#     --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
#     --query "data.sources[?contains(\"source-name\", 'OKE-${KUBERNETES_VERSION#v}-')].{name:\"source-name\",id:\"image-id\"}" \
#     --output table

# Pick an availability domain with A10 capacity.
# Iterate through ADs and use the first one with capacity available.
AD=""
for CANDIDATE in $(oci iam availability-domain list \
        --compartment-id "${OCI_COMPARTMENT_ID}" \
        --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
        --query 'data[].name' --raw-output | jq -r '.[]'); do
    AVAIL=$(oci limits resource-availability get \
        --compartment-id "${OCI_COMPARTMENT_ID}" \
        --service-name compute --limit-name gpu-a10-count \
        --availability-domain "${CANDIDATE}" \
        --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
        --query 'data.available' --raw-output 2>/dev/null)
    if [[ "${AVAIL}" =~ ^[0-9]+$ ]] && (( AVAIL > 0 )); then
        AD="${CANDIDATE}"
        echo "Selected AD with ${AVAIL} A10s available: ${AD}"
        break
    fi
done
[[ -z "${AD}" ]] && { echo "No AD with A10 capacity in ${OCI_REGION}"; exit 1; }

# CPU node pool (boot volume >= 100 GB for the router image)
oci ce node-pool create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --cluster-id "${CLUSTER_ID}" \
    --name "cpu-pool" \
    --kubernetes-version "${KUBERNETES_VERSION}" \
    --node-shape "VM.Standard.E5.Flex" \
    --node-shape-config '{"ocpus": 2, "memoryInGBs": 16}' \
    --node-image-id "${CPU_IMAGE_ID}" \
    --node-boot-volume-size-in-gbs 100 \
    --size 1 \
    --placement-configs "[{\"availabilityDomain\": \"${AD}\", \"subnetId\": \"${WORKER_SUBNET_ID}\"}]" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

# GPU node pool (boot volume 200 GB)
oci ce node-pool create \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --cluster-id "${CLUSTER_ID}" \
    --name "gpu-pool" \
    --kubernetes-version "${KUBERNETES_VERSION}" \
    --node-shape "VM.GPU.A10.1" \
    --node-image-id "${GPU_IMAGE_ID}" \
    --node-boot-volume-size-in-gbs 200 \
    --size 1 \
    --placement-configs "[{\"availabilityDomain\": \"${AD}\", \"subnetId\": \"${WORKER_SUBNET_ID}\"}]" \
    --initial-node-labels '[{"key": "app", "value": "gpu"}, {"key": "nvidia.com/gpu", "value": "true"}]' \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

Wait for both node pools to show nodes as ACTIVE (~10 minutes):

watch -n 30 "oci ce node-pool list \
    --compartment-id ${OCI_COMPARTMENT_ID} \
    --cluster-id ${CLUSTER_ID} \
    --profile ${OCI_PROFILE} --region ${OCI_REGION} \
    --query 'data[].{name:name,nodes:nodes[].{ip:\"private-ip\",state:\"lifecycle-state\"}}'"

Do not proceed to Step 6 until both node pools show nodes as ACTIVE.

Important: The CPU boot volume must be at least 100 GB. The vLLM router image is ~10.5 GB and the default 47 GB boot volume causes pod eviction.

Step 6: Connect to the private cluster

Download kubeconfig and configure for tunnel access:

oci ce cluster create-kubeconfig \
    --cluster-id "${CLUSTER_ID}" \
    --file ~/.kube/config-nemotron \
    --region "${OCI_REGION}" \
    --token-version 2.0.0 \
    --kube-endpoint PRIVATE_ENDPOINT \
    --profile "${OCI_PROFILE}" --overwrite

export KUBECONFIG=~/.kube/config-nemotron

# Get the private endpoint IP
PRIVATE_IP=$(oci ce cluster get --cluster-id "${CLUSTER_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data.endpoints."private-endpoint"' --raw-output | cut -d: -f1)

# Update kubeconfig to use localhost tunnel
CLUSTER_CTX=$(kubectl config view --minify -o jsonpath='{.clusters[0].name}')
kubectl config set-cluster "${CLUSTER_CTX}" \
    --server=https://127.0.0.1:6443 \
    --insecure-skip-tls-verify=true

If your OCI CLI profile is not DEFAULT, add it to the kubeconfig:

# In the users[].user.exec section, replace env: [] with:
env:
  - name: OCI_CLI_PROFILE
    value: YOUR_PROFILE

Create a Bastion session and start the SSH tunnel:

SESSION_ID=$(oci bastion session create-port-forwarding \
    --bastion-id "${BASTION_ID}" \
    --target-private-ip "${PRIVATE_IP}" \
    --target-port 6443 \
    --session-ttl 10800 \
    --display-name "nemotron-kubectl" \
    --ssh-public-key-file ~/.ssh/id_ed25519.pub \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query "data.id" --raw-output)

# Wait for session to become ACTIVE, then start tunnel
ssh -i ~/.ssh/id_ed25519 -N -L 6443:${PRIVATE_IP}:6443 \
    -p 22 -o StrictHostKeyChecking=no -o ServerAliveInterval=30 \
    ${SESSION_ID}@host.bastion.${OCI_REGION}.oci.oraclecloud.com &

# Verify
kubectl get nodes

Note: Bastion sessions expire after the TTL (default 3 hours). Create a new session and restart the tunnel when access drops.

Step 7: Expand boot volume filesystems

OCI boot volumes provision only ~47 GB of usable root filesystem regardless of the requested size. Both nodes must be expanded.

Why this matters: The vLLM engine image is ~10 GB, the router image is ~10.5 GB, and the model weights are ~16 GB. Without expansion, pods get evicted for low ephemeral storage.

For each node, run the following (use a unique pod name per node):

NODE_IP=<node-internal-ip>
POD_NAME=expand-$(echo $NODE_IP | tr '.' '-')

kubectl run ${POD_NAME} --restart=Never \
  --image=busybox:latest \
  --overrides="{
    \"spec\":{
      \"nodeName\":\"${NODE_IP}\",
      \"tolerations\":[{\"operator\":\"Exists\"}],
      \"containers\":[{
        \"name\":\"expand\",
        \"image\":\"busybox:latest\",
        \"command\":[\"sleep\",\"600\"],
        \"securityContext\":{\"privileged\":true},
        \"volumeMounts\":[{\"name\":\"host\",\"mountPath\":\"/host\"}]
      }],
      \"volumes\":[{\"name\":\"host\",\"hostPath\":{\"path\":\"/\"}}]
    }
  }"

kubectl wait --for=condition=Ready pod/${POD_NAME} --timeout=60s

kubectl exec ${POD_NAME} -- chroot /host bash -c '
  growpart /dev/sda 3
  sleep 3
  pvresize /dev/sda3
  lvextend -l +100%FREE /dev/ocivolume/root
  xfs_growfs /
  df -h /
'

kubectl delete pod ${POD_NAME} --force

Repeat for each node. Expected results:

GPU node (200 GB boot volume): 36 GB → ~189 GB usable
CPU node (100 GB boot volume): 36 GB → ~89 GB usable

Kubelet caches capacity at startup — in-place systemctl restart kubelet does not refresh it. See Step 7b.

Step 7b: Soft-reset each node so kubelet re-reads disk capacity

Drain each node, soft-reset the VM, wait for Ready, uncordon:

for NODE_IP in <cpu-node-ip> <gpu-node-ip>; do
    # Resolve the OCI instance OCID via the node's providerID, which OKE
    # sets to oci://<instance-ocid>. (`oci ce node-pool list` does not
    # populate the nested `nodes` array, so a list-based lookup returns
    # null; `get` per pool also works but is noisier.)
    INSTANCE_ID=$(kubectl get node "${NODE_IP}" \
        -o jsonpath='{.spec.providerID}' | sed 's|^oci://||')

    kubectl cordon "${NODE_IP}"
    kubectl drain "${NODE_IP}" --ignore-daemonsets --delete-emptydir-data \
        --force --grace-period=30 --timeout=120s || true

    oci compute instance action \
        --instance-id "${INSTANCE_ID}" --action SOFTRESET \
        --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

    # Wait for VM RUNNING, then for node Ready
    until [[ "$(oci compute instance get --instance-id "${INSTANCE_ID}" \
            --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
            --query 'data."lifecycle-state"' --raw-output)" == "RUNNING" ]]; do
        sleep 15
    done
    until kubectl get node "${NODE_IP}" \
            -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' \
            | grep -q True; do
        sleep 15
    done

    kubectl uncordon "${NODE_IP}"
done

Verify kubelet picked up the expanded capacity before continuing:

for NODE in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
    CAP=$(kubectl get node "${NODE}" \
        -o jsonpath='{.status.capacity.ephemeral-storage}')
    echo "${NODE}: ${CAP}"
done

Expected: CPU node ~93476416Ki (~89 GiB), GPU node ~198056192Ki (~189 GiB). If either still shows ~37206272Ki, rerun the soft-reset for that node.

Note: a node can report Ready slightly before kubelet republishes the new ephemeral-storage capacity, so the value may briefly still read ~37206272Ki right after uncordon. Re-check after ~60s before concluding a rerun is needed.

Step 8: Create StorageClasses

kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oci-block-storage-enc
provisioner: blockvolume.csi.oraclecloud.com
parameters:
  vpusPerGB: "10"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

Step 9: Patch CoreDNS for GPU tolerations

kubectl patch deployment coredns -n kube-system --type='json' \
  -p='[{"op":"add","path":"/spec/template/spec/tolerations/-",
        "value":{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}}]'

kubectl patch deployment kube-dns-autoscaler -n kube-system --type='json' \
  -p='[{"op":"add","path":"/spec/template/spec/tolerations/-",
        "value":{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}}]'

Step 10: Create the templates PVC

The vllm-stack chart (0.1.10) mounts a vllm-templates-pvc volume in every engine pod. This PVC must exist before deploying:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-templates-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: oci-block-storage-enc
  resources:
    requests:
      storage: 1Gi
EOF

Step 11: Deploy vLLM

The checked-in values file vllm_oke_phoenix_private_values.yaml contains the validated configuration for this deployment.

helm repo add vllm https://vllm-project.github.io/production-stack
helm repo update

helm upgrade --install vllm vllm/vllm-stack \
  -n default \
  -f vllm_oke_phoenix_private_values.yaml

Do not pass --wait to Helm. The engine pod takes several minutes to pull the image (~10 GB) and download the model.

Monitor progress:

kubectl get pods -n default -w

Wait for both pods to show 1/1 Running:

vllm-deployment-router-* — request router (CPU node)
vllm-llama31-nemotron-nano-8b-deployment-vllm-* — model engine (GPU node)

Step 12: Validate

kubectl -n default port-forward svc/vllm-router-service 8080:80

Health check:

curl -s http://127.0.0.1:8080/health
# {"status":"healthy"}

Model discovery:

curl -s http://127.0.0.1:8080/v1/models | jq .

Chat completion:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-8B-v1",
    "messages": [{"role": "user", "content": "Reply with NEMOTRON_OK"}]
  }'

Tool-calling smoke test:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-8B-v1",
    "messages": [{"role": "user", "content": "What time is it in UTC?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_utc_time",
        "description": "Return the current UTC time",
        "parameters": {"type": "object", "properties": {}, "required": []}
      }
    }]
  }'

Expected: finish_reason set to tool_calls.

Key vLLM settings

Setting	Value	Why
`tag`	`v0.19.0`	Pinned to validated vLLM version
`maxModelLen`	`4096`	Conservative context to fit single A10 (24 GB)
`gpuMemoryUtilization`	`0.95`	Maximize GPU memory for KV cache
`enableTool`	`true`	Enable tool / function calling
`toolCallParser`	`llama3_json`	Parser matching Nemotron's tool format
`extraArgs`	`--chat-template=...`	Template passed as CLI arg (chart's `chatTemplate` field prepends `/templates/`)
`storageClass`	`oci-block-storage-enc`	OCI Block Volume with balanced performance

Troubleshooting

Pods evicted for ephemeral storage

OCI boot volumes provision only ~47 GB of usable filesystem by default. Follow Step 7 to expand. If the boot volume itself is too small (default 47 GB), resize it first via the OCI CLI, then rescan the block device before running growpart:

echo 1 > /sys/class/block/sda/device/rescan

Engine pod evicted mid image pull despite Step 7 reporting success

Symptoms: engine pod reaches ContainerCreating, then kubelet evicts it with The node was low on resource: ephemeral-storage (or inodes), and FreeDiskSpaceFailed: ... but only found 0 bytes eligible to free.

Cause: kubelet's Node.Capacity.ephemeral-storage is cached at startup. Even after Step 7 expands the filesystem to ~189 GiB, kubelet continues to report the original ~37 GiB and triggers eviction thresholds against the stale value. Confirm with:

kubectl describe node <node-ip> | grep "ephemeral-storage:"

If the value is ~37206272Ki, apply Step 7b (soft-reset the VM). An in-place systemctl restart kubelet does not refresh the capacity.

SSH tunnel to OCI Bastion closes immediately after authentication

Symptoms: ssh -N -L 6443:... <session-id>@host.bastion.<region>.oci.oraclecloud.com completes publickey auth, reports Local forwarding listening on 127.0.0.1 port 6443, then: Connection to host.bastion.<region>.oci.oraclecloud.com closed by remote host. Port 6443 never stays open on the client.

Cause: OpenSSH 10.x (shipped on macOS 15+ and recent Linux distros) is incompatible with OCI Bastion's Go SSH server implementation for port-forwarding sessions.

Workaround: use the jump-host VM path in Appendix A. Downgrading the client to OpenSSH 9.x also works but is typically impractical on macOS.

Engine pod stays Pending with PVC not found

The vllm-stack chart (0.1.10) requires vllm-templates-pvc to exist before the engine pod can schedule. See Step 10.

Engine pod crashes with chat template error

The chart's chatTemplate field prepends /templates/ to the path. Pass the template via vllmConfig.extraArgs instead:

vllmConfig:
  extraArgs:
    - "--chat-template=/vllm-workspace/examples/tool_chat_template_llama3.1_json.jinja"

Tool calling does not work

Ensure all of these are set in the values file:

enableTool: true
toolCallParser: llama3_json
--chat-template=... in vllmConfig.extraArgs

`kubectl` cannot reach the cluster

Re-establish the Bastion tunnel. Sessions expire after the configured TTL.

Helm upgrade fails with field manager conflict

Uninstall and reinstall:

helm uninstall vllm -n default
helm install vllm vllm/vllm-stack -n default -f vllm_oke_phoenix_private_values.yaml

Cleanup

To tear down all resources:

# 1. Uninstall Helm release and PVCs
helm uninstall vllm -n default
kubectl delete pvc --all -n default

# 2. List and delete node pools
oci ce node-pool list --compartment-id "${OCI_COMPARTMENT_ID}" \
    --cluster-id "${CLUSTER_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data[].{name:name,id:id}' --output table

oci ce node-pool delete --node-pool-id <cpu-pool-id> --force \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"
oci ce node-pool delete --node-pool-id <gpu-pool-id> --force \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

# 3. Wait for node pools, then delete cluster
oci ce cluster delete --cluster-id "${CLUSTER_ID}" --force \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

# 4. Delete bastion
oci bastion bastion delete --bastion-id "${BASTION_ID}" --force \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

# 5. Wait for cluster deletion, then delete networking
#    Delete subnets first, then route tables, gateways, and VCN
for SUBNET_ID in "${API_SUBNET_ID}" "${WORKER_SUBNET_ID}" \
                  "${LB_SUBNET_ID}" "${BASTION_SUBNET_ID}"; do
    oci network subnet delete --subnet-id "${SUBNET_ID}" --force \
        --profile "${OCI_PROFILE}" --region "${OCI_REGION}"
done

# Delete non-default route tables, security lists, then gateways, then VCN

Alternative: Terraform

A Terraform sample using the oracle-terraform-modules/oke/oci module is available in terraform/ for reference. Note that the module's NSG configuration requires its built-in bastion compute host (create_bastion = true) for OCI Bastion port-forwarding to work. The manual CLI approach above is recommended for initial deployments.

Appendix A: Jump-host VM alternative (OpenSSH 10.x)

Use this when ssh -V reports OpenSSH 10.x. Replaces Step 4 and the bastion-session block in Step 6.

Trade-off: this is a public-IP VM, not OCI's managed bastion service. Terminate it during cleanup.

A.1 Launch the jump-host VM (replaces Step 4)

OL_IMAGE_ID=$(oci compute image list \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --operating-system "Oracle Linux" --operating-system-version "9" \
    --shape "VM.Standard.E5.Flex" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data[?"lifecycle-state"==`AVAILABLE`] | sort_by(@, &"time-created") | [-1].id' \
    --raw-output)

SSH_PUB=$(cat ~/.ssh/id_ed25519.pub)
METADATA=$(jq -cn --arg k "${SSH_PUB}" '{"ssh_authorized_keys": $k}')

oci compute instance launch \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --availability-domain "${AD}" \
    --display-name "${CLUSTER_NAME}-jumphost" \
    --shape "VM.Standard.E5.Flex" \
    --shape-config '{"ocpus":1,"memoryInGBs":8}' \
    --image-id "${OL_IMAGE_ID}" \
    --subnet-id "${BASTION_SUBNET_ID}" \
    --assign-public-ip true \
    --metadata "${METADATA}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --wait-for-state RUNNING

JUMP_HOST_ID=$(oci compute instance list \
    --compartment-id "${OCI_COMPARTMENT_ID}" \
    --display-name "${CLUSTER_NAME}-jumphost" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data[?"lifecycle-state"==`RUNNING`] | [0].id' --raw-output)

VNIC_ID=$(oci compute vnic-attachment list \
    --compartment-id "${OCI_COMPARTMENT_ID}" --instance-id "${JUMP_HOST_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data[0]."vnic-id"' --raw-output)

JUMP_HOST_IP=$(oci network vnic get --vnic-id "${VNIC_ID}" \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}" \
    --query 'data."public-ip"' --raw-output)

echo "Jump host public IP: ${JUMP_HOST_IP}"

# cloud-init may still be copying authorized_keys when the VM first reports
# RUNNING — wait for port 22 to accept connections before using ssh.
until nc -z -G 3 "${JUMP_HOST_IP}" 22 2>/dev/null; do sleep 2; done

A.2 Open the tunnel through the jump-host (replaces Step 6 bastion block)

Run Step 6 up through the kubectl config set-cluster server-URL rewrite, then skip the oci bastion session block and tunnel directly:

nohup ssh -f -N -L 6443:${PRIVATE_IP}:6443 \
    -i ~/.ssh/id_ed25519 \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -o IdentitiesOnly=yes -o ServerAliveInterval=30 \
    -o ExitOnForwardFailure=yes \
    opc@${JUMP_HOST_IP} < /dev/null > /tmp/nemotron-ssh-tunnel.log 2>&1

nc -z 127.0.0.1 6443 && echo "tunnel up" || echo "tunnel failed"
kubectl get nodes

No session TTL; restart the tunnel after a laptop sleep or network change.

A.3 Cleanup addition

When running the cleanup steps, also terminate the jump-host:

oci compute instance terminate --instance-id "${JUMP_HOST_ID}" --force \
    --preserve-boot-volume false \
    --profile "${OCI_PROFILE}" --region "${OCI_REGION}"

Navigation Menu

Uh oh!

Llama-3.1-Nemotron-Nano-8B-v1 on OCI OKE (Private Deployment)

Tested environment

Validated capabilities

Prerequisites

Architecture

Step 1: Set environment variables

Step 2: Create VCN and networking

Step 3: Create private OKE cluster

Step 4: Create OCI Bastion

Step 5: Create node pools

Step 6: Connect to the private cluster

Step 7: Expand boot volume filesystems

Step 7b: Soft-reset each node so kubelet re-reads disk capacity

Step 8: Create StorageClasses

Step 9: Patch CoreDNS for GPU tolerations

Step 10: Create the templates PVC

Step 11: Deploy vLLM

Step 12: Validate

Key vLLM settings

Troubleshooting

Pods evicted for ephemeral storage

Engine pod evicted mid image pull despite Step 7 reporting success

SSH tunnel to OCI Bastion closes immediately after authentication

Engine pod stays Pending with PVC not found

Engine pod crashes with chat template error

Tool calling does not work

`kubectl` cannot reach the cluster

Helm upgrade fails with field manager conflict

Cleanup

Alternative: Terraform

Appendix A: Jump-host VM alternative (OpenSSH 10.x)

A.1 Launch the jump-host VM (replaces Step 4)

A.2 Open the tunnel through the jump-host (replaces Step 6 bastion block)

A.3 Cleanup addition

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Llama-3.1-Nemotron-Nano-8B-v1 on OCI OKE (Private Deployment)

Tested environment

Validated capabilities

Prerequisites

Architecture

Step 1: Set environment variables

Step 2: Create VCN and networking

Step 3: Create private OKE cluster

Step 4: Create OCI Bastion

Step 5: Create node pools

Step 6: Connect to the private cluster

Step 7: Expand boot volume filesystems

Step 7b: Soft-reset each node so kubelet re-reads disk capacity

Step 8: Create StorageClasses

Step 9: Patch CoreDNS for GPU tolerations

Step 10: Create the templates PVC

Step 11: Deploy vLLM

Step 12: Validate

Key vLLM settings

Troubleshooting

Pods evicted for ephemeral storage

Engine pod evicted mid image pull despite Step 7 reporting success

SSH tunnel to OCI Bastion closes immediately after authentication

Engine pod stays Pending with PVC not found

Engine pod crashes with chat template error

Tool calling does not work

kubectl cannot reach the cluster

Helm upgrade fails with field manager conflict

Cleanup

Alternative: Terraform

Appendix A: Jump-host VM alternative (OpenSSH 10.x)

A.1 Launch the jump-host VM (replaces Step 4)

A.2 Open the tunnel through the jump-host (replaces Step 6 bastion block)

A.3 Cleanup addition

`kubectl` cannot reach the cluster