Skip to content

Commit 9e0cf02

Browse files
johnbelamaricjohnbelamaric
authored
Add DRA+CCC example (#1569)
* Add DRA+CCC example Signed-off-by: John Belamaric <jbelamaric@google.com> * Fix the table formatting * Add some results Signed-off-by: John Belamaric <jbelamaric@google.com> * Fix from Gemini review feedback * Some updates and tweaks for better demo'ing Signed-off-by: John Belamaric <jbelamaric@google.com> * Simplify to apply all k8s resources with one file Signed-off-by: John Belamaric <jbelamaric@google.com> * Update for latest docs and code * Fix missing volume mount * Updates from demo * Fix license header --------- Signed-off-by: John Belamaric <jbelamaric@google.com>
1 parent ae81bc4 commit 9e0cf02

4 files changed

Lines changed: 405 additions & 0 deletions

File tree

‎autoscaling/custom-compute-classes/dra/README.md‎

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
# DRA and Custom Compute Class Examples
2+
3+
Custom Compute Classes (CCC) offer a mechanism to manage autoscaling of nodes.
4+
With CCC, the cluster administrator can put nodes into classes designed to meet
5+
specific workload needs. By using classes rather than specific node pools, the
6+
administrator has flexibility to meet the workload needs with different types of
7+
nodes. This can allow autoscaling that prefers spot instances or node shapes for
8+
which the administrator has reservations.
9+
10+
For workloads that utilize specialized devices like GPUs and TPUs, you cannot
11+
always swap out one node shape for another, without adjusting the resource
12+
requirements of the Pod. For example, if a Pod uses the Device Plugin extended
13+
resources to request 2 GPUs (`nvidia.com/gpu: 2`), supplying that Pod with a
14+
node that has a single, larger GPU wouldn't work without updating that Pod spec.
15+
16+
Dynamic Resource Allocation (DRA) is a beta feature in Kubernetes 1.32 which
17+
allows workload authors to more flexibly specify the demands of their workload.
18+
This enables workload authors to write their specifications once, while giving
19+
cluster administrators more options to meet those needs.
20+
21+
## One compute class, many type of GPU nodes
22+
23+
For example, consider an inference workload that has modest latency
24+
requirements, and needs 24GB of GPU memory, 1 vCPU, and 8GB of system memory.
25+
There are a wide variety of node shapes that could run this workload. The
26+
minimum sized shapes with various NVIDIA GPU models that qualify, in the admin's
27+
preferred order, are:
28+
29+
| Node Shape | vCPUS | Mem (GB) | GPUs | GPU Mem (GB) |
30+
| -------------------|-------|----------|----------|--------------|
31+
| n1-standard-4+T4 | 4 | 15 | 2 x T4 | 32 |
32+
| g2-standard-4 | 4 | 16 | 1 x L4 | 24 |
33+
| n1-standard-4+P4 | 4 | 15 | 4 x P4 | 32 |
34+
| n1-standard-4+P100 | 4 | 15 | 2 x P100 | 32 |
35+
| n1-standard-4+V100 | 4 | 15 | 2 x V100 | 32 |
36+
| a2-highgpu-1g | 12 | 85 | 1 x A100 | 40 |
37+
| a3-highgpu-1g | 26 | 234 | 1 x H100 | 80 |
38+
39+
Given that the A2 and A3 machines are substantially overpowered for this
40+
workload, the cluster admin would prefer that they are not used in this case.
41+
Therefore, they could build a compute class that allows any of the N1 or G2
42+
machines, but prefers them in the order given, for cost savings. It's also
43+
possible to use spot instances for any of these node pools.
44+
45+
Notice though that some of these machines have 1 GPU, some have 2, and one shape
46+
even has 4 GPUs. Using the current Kubernetes Device Plugin, we would have to
47+
change our `requests` and `limits` for the Pod's `nvidia.com/gpu` extended
48+
resource, or the Pod would not have access to the right number of GPUs, or may
49+
not even schedule.
50+
51+
This is where DRA comes in. Rather than specifying the exact count of GPUs, DRA
52+
allows you to ask for `All` GPUs on the node. If the Pod lands on a node with
53+
one GPU, then that one GPU will be mounted into the Pod; if it lands on one with
54+
4 GPUs, then all 4 will be mounted into the Pod. This flexibility allows the
55+
workload author to create a single Deployment, and for CCC to base autoscaling
56+
on that deployment.
57+
58+
Of course, the workload needs to automatically discover the number and type of
59+
GPU available to it, and make use of them all. But that is a common behavior of many
60+
existing workloads, as long as they are all NVIDIA GPUs.
61+
62+
Let's give this a try. In this example, we will create a compute class that
63+
allows the T4, L4, or P4 options, in that order of priority. We will make the P4
64+
option use spot VMs, to reduce costs.
65+
66+
First, we need a GKE cluster with the DRA beta enabled. This is available
67+
starting in GKE 1.32. For this example, we will use 1.33:
68+
69+
```console
70+
CLUSTER_NAME=drabeta
71+
LOCATION=us-west4
72+
VERSION=1.33
73+
gcloud container clusters create \
74+
--location ${LOCATION} \
75+
--release-channel rapid \
76+
--cluster-version ${VERSION} \
77+
--enable-kubernetes-unstable-apis=resource.k8s.io/v1beta1/deviceclasses,resource.k8s.io/v1beta1/resourceclaims,resource.k8s.io/v1beta1/resourceclaimtemplates,resource.k8s.io/v1beta1/resourceslices \
78+
${CLUSTER_NAME}
79+
```
80+
81+
When we create node pools in this example, we will use Ubuntu nodes and we will
82+
manually install the NVIDIA GPU drivers, the NVIDIA Container Toolkit, and the
83+
NVIDIA DRA Driver for GPUs. These are all the components we need to use DRA.
84+
85+
Installing the right NVIDIA drivers for Ubuntu nodes is described in the Google
86+
Cloud GPU
87+
[documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#ubuntu).
88+
For the GPUs we will use in this example, the following command is sufficient:
89+
90+
```console
91+
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/ubuntu/daemonset-preloaded.yaml
92+
```
93+
94+
The container toolkit is installed via a DaemonSet:
95+
96+
```console
97+
kubectl apply -f nvidia-container-toolkit-installer.yaml
98+
```
99+
100+
The DRA driver is installed via a Helm chart, as described in the Google Cloud
101+
[documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/set-up-dra#install).
102+
103+
Next, we need node pools representing each machine type. As of now, we cannot use
104+
the Node Autoprovisioning feature of CCC with DRA, because of the specialized
105+
labels needed to switch to the DRA driver. Let's create a node pool for each
106+
machine type. We will create them with a max of three nodes, so we can test the
107+
prioritization functionality without using too many resources:
108+
109+
```console
110+
COMPUTE_CLASS=inference-1x8x24
111+
# N1 with 2xT4
112+
gcloud container node-pools create "n1-standard-4-2xt4" \
113+
--cluster "${CLUSTER_NAME}" \
114+
--location "${LOCATION}" \
115+
--node-version "${VERSION}" \
116+
--machine-type "n1-standard-4" \
117+
--accelerator "type=nvidia-tesla-t4,count=2,gpu-driver-version=disabled" \
118+
--image-type "UBUNTU_CONTAINERD" \
119+
--num-nodes "0" \
120+
--enable-autoscaling \
121+
--min-nodes "0" \
122+
--max-nodes "3" \
123+
--node-labels=cloud.google.com/compute-class=${COMPUTE_CLASS},gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true \
124+
--node-taints=cloud.google.com/compute-class=${COMPUTE_CLASS}:NoSchedule
125+
126+
# G2 with 1xL4
127+
gcloud container node-pools create "g2-standard-4" \
128+
--cluster "${CLUSTER_NAME}" \
129+
--location "${LOCATION}" \
130+
--node-version "${VERSION}" \
131+
--machine-type "g2-standard-4" \
132+
--accelerator "type=nvidia-l4,gpu-driver-version=disabled" \
133+
--image-type "UBUNTU_CONTAINERD" \
134+
--num-nodes "0" \
135+
--enable-autoscaling \
136+
--min-nodes "0" \
137+
--max-nodes "3" \
138+
--node-labels=cloud.google.com/compute-class=${COMPUTE_CLASS},gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true \
139+
--node-taints=cloud.google.com/compute-class=${COMPUTE_CLASS}:NoSchedule
140+
141+
# N1 with 4xP4 spot instances
142+
gcloud container node-pools create "n1-standard-4-4xp4" \
143+
--cluster "${CLUSTER_NAME}" \
144+
--location "${LOCATION}" \
145+
--node-version "${VERSION}" \
146+
--machine-type "n1-standard-4" \
147+
--accelerator "type=nvidia-tesla-p4,count=4,gpu-driver-version=disabled" \
148+
--image-type "UBUNTU_CONTAINERD" \
149+
--num-nodes "0" \
150+
--enable-autoscaling \
151+
--min-nodes "0" \
152+
--max-nodes "3" \
153+
--spot \
154+
--node-labels=cloud.google.com/compute-class=${COMPUTE_CLASS},gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true \
155+
--node-taints=cloud.google.com/compute-class=${COMPUTE_CLASS}:NoSchedule
156+
```
157+
158+
Once we have those node pools defined, we need to create a custom compute class
159+
that prioritizes them in the lowest-cost first order.
160+
161+
```yaml
162+
apiVersion: cloud.google.com/v1
163+
kind: ComputeClass
164+
metadata:
165+
name: inference-1x8x24
166+
spec:
167+
priorities:
168+
- nodepools: [n1-standard-4-2xt4]
169+
- nodepools: [g2-standard-4]
170+
- nodepools: [n1-standard-4-4xp4]
171+
whenUnsatisfiable: DoNotScaleUp
172+
---
173+
174+
```
175+
176+
Now, deploy a workload. We will start with a single replica, which will get
177+
scheduled to one of the nodes we already created above. The example creates a
178+
ResourceClaimTemplate that asks for all GPUs on the node, and a Deployment with
179+
a Pod template references that ResourceClaimTemplate. For each Pod created by
180+
the Deployment, a new ResourceClaim will be created based on the
181+
ResourceClaimTemplate, and associated with that new Pod. This will allow
182+
scheduling of the Pod to any node with GPUs, and all the GPUs on that node will
183+
be assigned to the Pod and available to it. To illustrate this, the example uses
184+
`nvidia-smi` to print out all the GPUs it sees.
185+
186+
You can apply this workload with `kubectl apply -f deployment.yaml`.
187+
188+
Since at present CA does not understand the DRA resources, it won't trigger a
189+
scale up because of insufficient GPU resources. This is in development and will
190+
be resolved soon. Until then, we use an anti-affinity rule to force
191+
one-pod-per-node and trigger scaling; you will see this if you examine
192+
`deployment.yaml`.
193+
194+
Results of scaling to 4 replicas:
195+
196+
```console
197+
$ k get po
198+
NAME READY STATUS RESTARTS AGE
199+
ccc-gpu-6b6866cb68-66zrl 1/1 Running 0 29s
200+
ccc-gpu-6b6866cb68-prb2k 1/1 Running 0 29s
201+
ccc-gpu-6b6866cb68-rj7z6 1/1 Running 0 29s
202+
ccc-gpu-6b6866cb68-t2jrp 1/1 Running 0 29s
203+
$ k logs ccc-gpu-6b6866cb68-prb2k
204+
GPU 0: Tesla T4 (UUID: GPU-9cbaabad-b724-9242-39e2-486a9450527a)
205+
GPU 1: Tesla T4 (UUID: GPU-3d424488-8672-f8af-3fb7-2ace22ec966e)
206+
$ k logs ccc-gpu-6b6866cb68-66zrl
207+
GPU 0: Tesla T4 (UUID: GPU-3901c076-1e26-f103-4dcd-b3a8d5723bcf)
208+
GPU 1: Tesla T4 (UUID: GPU-4f6e00e5-3d4a-b762-9128-242138949d6f)
209+
$ k logs ccc-gpu-6b6866cb68-rj7z6
210+
GPU 0: Tesla T4 (UUID: GPU-b8d5c02e-d43a-82b4-9219-8d3920fedb0c)
211+
GPU 1: Tesla T4 (UUID: GPU-59057b0c-944b-a66c-7926-89b4cc6c42c8)
212+
$ k logs ccc-gpu-6b6866cb68-t2jrp
213+
GPU 0: NVIDIA L4 (UUID: GPU-51d13b1f-8fe5-9270-be7d-2561dadad56e)
214+
$ k get nodes
215+
NAME STATUS ROLES AGE VERSION
216+
gke-drabeta-default-pool-64569529-32z4 Ready <none> 26h
217+
v1.32.0-gke.1358000
218+
gke-drabeta-default-pool-64569529-3wmn Ready <none> 26h
219+
v1.32.0-gke.1358000
220+
gke-drabeta-default-pool-64569529-57b6 Ready <none> 26h
221+
v1.32.0-gke.1358000
222+
gke-drabeta-g2-standard-4-68897633-vtd8 Ready <none> 25m
223+
v1.32.0-gke.1358000
224+
gke-drabeta-n1-standard-4-2xt4-a48308e9-45vl Ready <none> 25m
225+
v1.32.0-gke.1358000
226+
gke-drabeta-n1-standard-4-2xt4-a48308e9-92pw Ready <none> 28m
227+
v1.32.0-gke.1358000
228+
gke-drabeta-n1-standard-4-2xt4-a48308e9-dnqw Ready <none> 25m
229+
v1.32.0-gke.1358000
230+
```

‎autoscaling/custom-compute-classes/dra/ccc-inference-1x8x24.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Copyright 2025 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
apiVersion: cloud.google.com/v1
16+
kind: ComputeClass
17+
metadata:
18+
name: inference-1x8x24
19+
spec:
20+
priorities:
21+
- nodepools: [n1-standard-4-2xt4]
22+
- nodepools: [g2-standard-4]
23+
- nodepools: [n1-standard-4-4xp4]
24+
whenUnsatisfiable: DoNotScaleUp

‎autoscaling/custom-compute-classes/dra/deployment.yaml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Copyright 2025 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
apiVersion: resource.k8s.io/v1beta1
16+
kind: ResourceClaimTemplate
17+
metadata:
18+
name: all-gpus
19+
spec:
20+
spec:
21+
devices:
22+
requests:
23+
- name: gpu
24+
deviceClassName: gpu.nvidia.com
25+
allocationMode: All
26+
---
27+
apiVersion: apps/v1
28+
kind: Deployment
29+
metadata:
30+
name: ccc-gpu
31+
spec:
32+
replicas: 1
33+
selector:
34+
matchLabels:
35+
app: ccc-gpu
36+
template:
37+
metadata:
38+
labels:
39+
app: ccc-gpu
40+
spec:
41+
nodeSelector:
42+
cloud.google.com/compute-class: inference-1x8x24
43+
containers:
44+
- name: ctr
45+
image: ubuntu:22.04
46+
command: ["bash", "-c"]
47+
args: ["while [ 1 ]; do date; /host/opt/nvidia/bin/nvidia-smi -L || echo Waiting...; sleep 30; done"]
48+
resources:
49+
claims:
50+
- name: gpu
51+
volumeMounts:
52+
- name: host-root
53+
mountPath: /host
54+
volumes:
55+
- name: host-root
56+
hostPath:
57+
path: /
58+
resourceClaims:
59+
- name: gpu
60+
resourceClaimTemplateName: all-gpus
61+
tolerations:
62+
- key: "nvidia.com/gpu"
63+
operator: "Exists"
64+
effect: "NoSchedule"
65+
affinity:
66+
podAntiAffinity:
67+
requiredDuringSchedulingIgnoredDuringExecution:
68+
- labelSelector:
69+
matchExpressions:
70+
- key: app
71+
operator: In
72+
values:
73+
- ccc-gpu
74+
topologyKey: "kubernetes.io/hostname"
75+

0 commit comments

Comments
 (0)