Skip to content

Commit 83f3e51

Browse files
haoqing0110haoqing0110
committed
update kueue-addon readme with Design Details and Workflow
Signed-off-by: Qing Hao <qhao@redhat.com>
1 parent 7a432ce commit 83f3e51

3 files changed

Lines changed: 125 additions & 6 deletions

File tree

‎kueue-addon/README.md‎

Lines changed: 123 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,6 @@ The whole setup steps about this solution, please refer to this [Kueue Integrati
5252

5353
## Installation
5454

55-
5655
On the hub cluster, deploy the addon.
5756

5857
```bash
@@ -62,8 +61,17 @@ make deploy
6261
You can install the addons via the helm charts.
6362

6463
```bash
65-
helm install kueue-addon charts/kueue-addon/ \
66-
-n open-cluster-management-addon --create-namespace
64+
$ helm repo add ocm https://open-cluster-management.io/helm-charts/
65+
$ helm repo update
66+
$ helm search repo ocm/kueue-addon
67+
NAME CHART VERSION APP VERSION DESCRIPTION
68+
ocm/kueue-addon <chart-version> <app-version> A Helm chart for Open Cluster Management Kueue ...
69+
$ helm install \
70+
-n open-cluster-management-addon --create-namespace \
71+
kueue-addon ocm/kueue-addon
72+
# Uncomment the following lines to customize your installation:
73+
# --set skipClusterSetBinding=true \
74+
# --set image.tag=<chart-version> \
6775
```
6876

6977
To confirm the installation from hub:
@@ -100,4 +108,115 @@ default-flavor 4h28m
100108

101109
## Usage
102110

103-
The usage please refer to this [Kueue Integration Solution](https://github.com/open-cluster-management-io/ocm/blob/main/solutions/kueue-admission-check).
111+
The usage please refer to this [Kueue Integration Solution](https://github.com/open-cluster-management-io/ocm/blob/main/solutions/kueue-admission-check).
112+
113+
## Design Details and Workflow
114+
115+
### OCM Admission Check Controller
116+
117+
The OCM Admission Check Controller will integrate OCM `Placement` results into MultiKueue by reading `PlacementDecisions` and generating the necessary `MultiKueueConfig` and `MultiKueueCluster` resources.
118+
119+
- `controllerName`: Identifies the controller that processes the Admission Check, currently set to `open-cluster-management.io/placement`
120+
- `parameters`: Identifies a configuration with additional parameters for the check, here we add the existing OCM resource `Placement`. Clusters specified in the `Placement` will be bound to the `kueue-system` namespace.
121+
122+
Example OCM AdmissionCheck CR:
123+
124+
```yaml
125+
# OCM implements an admissioncheck controller to automate the MultiKueue setup process.
126+
# Leverages OCM's placement mechanism to select clusters based on specific criteria.
127+
# MultiKueueConfigs and MultiKueueClusters are generated dynamically based on OCM placement decisions.
128+
apiVersion: kueue.x-k8s.io/v1beta1
129+
kind: AdmissionCheck
130+
metadata:
131+
name: multikueue-config-demo2
132+
spec:
133+
controllerName: open-cluster-management.io/placement
134+
parameters:
135+
apiGroup: cluster.open-cluster-management.io
136+
kind: Placement
137+
name: multikueue-config-demo2
138+
```
139+
### Configuration Process: Before and After OCM Admission Check Controller
140+
141+
**Before:**
142+
143+
Admins manually create both `MultiKueueConfig` (listing clusters) and a `MultiKueueCluster` (with kubeconfig secret) for each cluster.
144+
145+
```yaml
146+
apiVersion: kueue.x-k8s.io/v1beta1
147+
kind: MultiKueueConfig
148+
metadata:
149+
name: multikueue-config-demo1
150+
spec:
151+
clusters:
152+
- multikueue-config-demo1-cluster1
153+
- multikueue-config-demo1-cluster2
154+
---
155+
apiVersion: kueue.x-k8s.io/v1beta1
156+
kind: MultiKueueCluster
157+
metadata:
158+
name: multikueue-config-demo1-cluster1
159+
spec:
160+
kubeConfig:
161+
locationType: Secret
162+
location: multikueue-cluster1
163+
---
164+
apiVersion: kueue.x-k8s.io/v1beta1
165+
kind: MultiKueueCluster
166+
metadata:
167+
name: multikueue-config-demo1-cluster2
168+
spec:
169+
kubeConfig:
170+
locationType: Secret
171+
location: multikueue-cluster2
172+
```
173+
174+
**After:**
175+
176+
Admins only need to add `AdmissionChecks` to the `ClusterQueue`. The controller automates `MultiKueueConfig` and `MultiKueueCluster` creation based on `Placement` decisions.
177+
178+
```yaml
179+
apiVersion: kueue.x-k8s.io/v1beta1
180+
kind: ClusterQueue
181+
metadata:
182+
name: "cluster-queue"
183+
spec:
184+
...
185+
admissionChecks:
186+
- multikueue-demo2
187+
- multikueue-config-demo2
188+
---
189+
apiVersion: kueue.x-k8s.io/v1beta1
190+
kind: AdmissionCheck
191+
metadata:
192+
name: multikueue-demo2
193+
spec:
194+
controllerName: kueue.x-k8s.io/multikueue
195+
parameters:
196+
apiGroup: kueue.x-k8s.io
197+
kind: MultiKueueConfig
198+
name: multikueue-config-demo2
199+
---
200+
apiVersion: kueue.x-k8s.io/v1beta1
201+
kind: AdmissionCheck
202+
metadata:
203+
name: multikueue-config-demo2
204+
spec:
205+
controllerName: open-cluster-management.io/placement
206+
parameters:
207+
apiGroup: cluster.open-cluster-management.io
208+
kind: Placement
209+
name: multikueue-config-demo2
210+
```
211+
212+
This automation greatly reduces manual effort and ensures that MultiKueue environments always reflect the latest OCM Placement decisions, supporting advanced scheduling scenarios such as label-based selection and dynamic scoring with `AddonPlacementScore`.
213+
214+
### Workflow
215+
216+
- When kueue-addon is intalled, the credential controller generates kubeconfig secrets of each cluster for MultiKueue under `kueue-system` namespace.
217+
- User configure the `ClusterQueue` with `AdmissionCheck` and creates the `AdmissionCheck` and `Placement` resources.
218+
- OCM generates `PlacementDecision` when `Placement` created.
219+
- The admission check controller watches for `AdmissionCheck` resources referencing OCM `Placement`.
220+
- The admission check controller watches the `PlacementDecision`, creates or updates `MultiKueueCluster` resources with the kubeconfig details for each cluster, and also update these clusters in the `MultiKueueConfig` resource.
221+
- Finally, admission check controller updates the `AdmissionCheck` condition to true, indicating successful generation of the `MultiKueueConfig` and `MultiKueueCluster`, readying the [MultiKueue](https://kueue.sigs.k8s.io/docs/concepts/multikueue/) environment for job scheduling.
222+

‎kueue-addon/charts/kueue-addon/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ apiVersion: v2
22
name: kueue-addon
33
description: A Helm chart for Open Cluster Management Kueue Addon
44
type: application
5-
version: 0.1.1
5+
version: 0.1.2
66
appVersion: "0.9.0"
77
maintainers:
88
- name: haoqing0110

‎kueue-addon/charts/kueue-addon/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# Image configuration
55
image:
66
repository: quay.io/open-cluster-management/kueue-addon
7-
tag: latest
7+
tag:
88

99
# Skip cluster set binding
1010
skipClusterSetBinding: false

0 commit comments

Comments
 (0)