Skip to content

Commit 4db7b16

Browse files
haoqing0110haoqing0110
committed
install kueue by operator
Signed-off-by: Qing Hao <qhao@redhat.com>
1 parent 19bb240 commit 4db7b16

6 files changed

Lines changed: 193 additions & 9 deletions

File tree

‎kueue-addon/README.md‎

Lines changed: 61 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,14 @@ This controller is running on the hub, contains a credential controller and an a
3535
### Addon chart
3636
- **Addon deployment:** Deploy [Kueue addon controllers](#kueue-addon-controller) on the hub.
3737
- **Addon Template:** To deploy `ResourceFlavor`, `ClusterQueue` and `LocalQueue` resources need by MultiKueue to spoke clusters.
38-
- **Other addon files:** `ClusterManagementAddOn`, `ClusterRole`, `ClusterRoleBinding`, `ManagedClusterSetBinding`, `Placement` etc.
38+
- **Kueue Operator Template:** Optional operator-based Kueue installation spoke clusters.
39+
- **Other addon files:** `ClusterManagementAddOn`, `ClusterRole`, `ClusterRoleBinding`, `ManagedClusterSetBinding`, `Placement` etc.
3940

4041
## Prerequisites
4142

4243
- Open Cluster Management (OCM) installed with the following addons:
4344
- [Cluster Permission Addon](https://github.com/open-cluster-management-io/cluster-permission)
4445
- [Managed Service Account Addon](https://github.com/open-cluster-management-io/managed-serviceaccount)
45-
- Kueue already installed on the hub and spoke cluster
46-
- MultiKueue enabled on the hub
4746

4847
## Quick Start
4948

@@ -52,13 +51,27 @@ The whole setup steps about this solution, please refer to this [Kueue Integrati
5251

5352
## Installation
5453

55-
On the hub cluster, deploy the addon.
54+
### Method 1: Using Makefile
55+
56+
On the hub cluster, deploy the addon:
5657

5758
```bash
5859
make deploy
5960
```
6061

61-
You can install the addons via the helm charts.
62+
### Method 2: Using Helm Charts
63+
64+
The addon supports two installation approaches:
65+
66+
**Option 1: Manual Kueue Installation**
67+
- Kueue already installed on the hub and spoke cluster
68+
69+
**Option 2: Operator-based Installation**
70+
- [Kueue operator](https://github.com/openshift/kueue-operator) available in your cluster
71+
72+
#### Option 1: Manual Kueue Installation
73+
74+
Install the addon with manually pre-installed Kueue:
6275

6376
```bash
6477
$ helm repo add ocm https://open-cluster-management.io/helm-charts/
@@ -74,6 +87,49 @@ $ helm install \
7487
# --set image.tag=<chart-version> \
7588
```
7689

90+
#### Option 2: Operator-based Kueue Installation
91+
92+
Install the addon with automatic Kueue operator deployment:
93+
94+
Prepare a values.override.yaml with below content:
95+
96+
```yaml
97+
# Install Kueue via Operator (OpenShift/downstream only)
98+
installKueueViaOperator: true
99+
100+
# Operator Lifecycle Manager RBAC configuration
101+
operatorLifecycleManager:
102+
clusterRoleBindingName: kueue-operator-lifecycle-manager-rolebinding
103+
clusterRoleName: system:controller:operator-lifecycle-manager
104+
105+
# Kueue Operator configuration
106+
kueueOperator:
107+
name: kueue-operator
108+
namespace: openshift-kueue-operator
109+
operatorGroupName: openshift-kueue-operator
110+
channel: stable-v1.0
111+
source: redhat-operators
112+
sourceNamespace: openshift-marketplace
113+
startingCSV: kueue-operator.v1.0.0
114+
115+
# Cert Manager Operator configuration
116+
certManagerOperator:
117+
name: openshift-cert-manager-operator
118+
namespace: cert-manager-operator
119+
operatorGroupName: cert-manager-operator
120+
channel: stable-v1
121+
source: redhat-operators
122+
sourceNamespace: openshift-marketplace
123+
startingCSV: cert-manager-operator.v1.16.1
124+
```
125+
126+
```bash
127+
$ helm install \
128+
-n open-cluster-management-addon --create-namespace \
129+
-f values.override.yaml \
130+
kueue-addon ocm/kueue-addon
131+
```
132+
77133
To confirm the installation from hub:
78134

79135
```bash
@@ -219,4 +275,3 @@ This automation greatly reduces manual effort and ensures that MultiKueue enviro
219275
- The admission check controller watches for `AdmissionCheck` resources referencing OCM `Placement`.
220276
- The admission check controller watches the `PlacementDecision`, creates or updates `MultiKueueCluster` resources with the kubeconfig details for each cluster, and also update these clusters in the `MultiKueueConfig` resource.
221277
- Finally, admission check controller updates the `AdmissionCheck` condition to true, indicating successful generation of the `MultiKueueConfig` and `MultiKueueCluster`, readying the [MultiKueue](https://kueue.sigs.k8s.io/docs/concepts/multikueue/) environment for job scheduling.
222-

‎kueue-addon/charts/kueue-addon/templates/_operator.tpl‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
{{/*
2+
Operator Resources - Single Template
3+
*/}}
4+
{{- define "kueue-addon.operatorResources" -}}
5+
{{- if and .Values.kueueOperator .Values.certManagerOperator .Values.operatorLifecycleManager }}
6+
- apiVersion: rbac.authorization.k8s.io/v1
7+
kind: ClusterRoleBinding
8+
metadata:
9+
name: {{ .Values.operatorLifecycleManager.clusterRoleBindingName }}
10+
roleRef:
11+
apiGroup: rbac.authorization.k8s.io
12+
kind: ClusterRole
13+
name: {{ .Values.operatorLifecycleManager.clusterRoleName }}
14+
subjects:
15+
- kind: ServiceAccount
16+
name: klusterlet-work-sa
17+
namespace: open-cluster-management-agent
18+
- apiVersion: v1
19+
kind: Namespace
20+
metadata:
21+
name: {{ .Values.kueueOperator.namespace }}
22+
- apiVersion: operators.coreos.com/v1
23+
kind: OperatorGroup
24+
metadata:
25+
name: {{ .Values.kueueOperator.operatorGroupName }}
26+
namespace: {{ .Values.kueueOperator.namespace }}
27+
spec:
28+
upgradeStrategy: Default
29+
- apiVersion: operators.coreos.com/v1alpha1
30+
kind: Subscription
31+
metadata:
32+
name: {{ .Values.kueueOperator.name }}
33+
namespace: {{ .Values.kueueOperator.namespace }}
34+
spec:
35+
channel: {{ .Values.kueueOperator.channel }}
36+
installPlanApproval: Automatic
37+
name: {{ .Values.kueueOperator.name }}
38+
source: {{ .Values.kueueOperator.source }}
39+
sourceNamespace: {{ .Values.kueueOperator.sourceNamespace }}
40+
startingCSV: {{ .Values.kueueOperator.startingCSV }}
41+
- apiVersion: kueue.openshift.io/v1
42+
kind: Kueue
43+
metadata:
44+
name: cluster
45+
labels:
46+
app.kubernetes.io/managed-by: kustomize
47+
app.kubernetes.io/name: kueue-operator
48+
spec:
49+
config:
50+
integrations:
51+
frameworks:
52+
- BatchJob
53+
managementState: Managed
54+
- apiVersion: v1
55+
kind: Namespace
56+
metadata:
57+
name: {{ .Values.certManagerOperator.namespace }}
58+
- apiVersion: operators.coreos.com/v1
59+
kind: OperatorGroup
60+
metadata:
61+
name: {{ .Values.certManagerOperator.operatorGroupName }}
62+
namespace: {{ .Values.certManagerOperator.namespace }}
63+
spec:
64+
upgradeStrategy: Default
65+
- apiVersion: operators.coreos.com/v1alpha1
66+
kind: Subscription
67+
metadata:
68+
name: {{ .Values.certManagerOperator.name }}
69+
namespace: {{ .Values.certManagerOperator.namespace }}
70+
spec:
71+
channel: {{ .Values.certManagerOperator.channel }}
72+
installPlanApproval: Automatic
73+
name: {{ .Values.certManagerOperator.name }}
74+
source: {{ .Values.certManagerOperator.source }}
75+
sourceNamespace: {{ .Values.certManagerOperator.sourceNamespace }}
76+
startingCSV: {{ .Values.certManagerOperator.startingCSV }}
77+
{{- end }}
78+
{{- end }}

‎kueue-addon/charts/kueue-addon/templates/addon-template.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,7 @@ spec:
6262
name: {{ .Values.localQueue.name}}
6363
spec:
6464
clusterQueue: {{ .Values.clusterQueue.name}}
65+
{{- if .Values.installKueueViaOperator }}
66+
{{- include "kueue-addon.operatorResources" . | nindent 8 }}
67+
{{- end }}
68+

‎kueue-addon/charts/kueue-addon/templates/deployment.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ spec:
2121
args:
2222
- "/kueue-addon-controller"
2323
- "hub"
24+
env:
25+
- name: KUEUE_NAMESPACE
26+
value: {{ .Values.kueue.namespace }}
2427

2528
---
2629

‎kueue-addon/charts/kueue-addon/values.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,36 @@ clusterQueue:
3838
localQueue:
3939
name: user-queue
4040
namespace: default
41+
42+
# This is the namespace where the kueue controller is installed, will copy the multikueue secret to this namespace
43+
# Default is kueue-system
44+
kueue:
45+
namespace: "kueue-system"
46+
47+
# Install Kueue via Operator
48+
installKueueViaOperator: false
49+
50+
# Operator Lifecycle Manager RBAC configuration (uncomment when needed)
51+
# operatorLifecycleManager:
52+
# clusterRoleBindingName: <cluster-role-binding-name>
53+
# clusterRoleName: <cluster-role-name>
54+
55+
# Kueue Operator configuration (uncomment when needed)
56+
# kueueOperator:
57+
# name: <operator-name>
58+
# namespace: <operator-namespace>
59+
# operatorGroupName: <operator-group-name>
60+
# channel: <channel-version>
61+
# source: <operator-source>
62+
# sourceNamespace: <source-namespace>
63+
# startingCSV: <starting-csv>
64+
65+
# Cert Manager Operator configuration (uncomment when needed)
66+
# certManagerOperator:
67+
# name: <cert-manager-operator-name>
68+
# namespace: <cert-manager-namespace>
69+
# operatorGroupName: <cert-manager-operator-group>
70+
# channel: <cert-manager-channel>
71+
# source: <cert-manager-source>
72+
# sourceNamespace: <cert-manager-source-namespace>
73+
# startingCSV: <cert-manager-starting-csv>

‎kueue-addon/pkg/hub/controllers/common/value.go‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,29 @@
11
package common
22

3-
import "fmt"
3+
import (
4+
"fmt"
5+
"os"
6+
)
47

5-
const (
8+
var (
69
// MultiKueueResourceName is the name used for MultiKueue-related secrets
710
MultiKueueResourceName = "multikueue"
811

912
// KueueNamespace is the namespace where Kueue components are deployed
10-
KueueNamespace = "kueue-system"
13+
KueueNamespace = getKueueNamespace()
1114

1215
// AdmissionCheckControllerName is the name of the admission check controller
1316
AdmissionCheckControllerName = "open-cluster-management.io/placement"
1417
)
1518

19+
func getKueueNamespace() string {
20+
ns := os.Getenv("KUEUE_NAMESPACE")
21+
if ns == "" {
22+
return "kueue-system"
23+
}
24+
return ns
25+
}
26+
1627
// Get MultiKueue Kubeconfig Secret name
1728
func GetMultiKueueSecretName(clusterName string) string {
1829
return fmt.Sprintf("%s-%s", MultiKueueResourceName, clusterName)

0 commit comments

Comments
 (0)