install kueue by operator (open-cluster-management-io#54)

Signed-off-by: Qing Hao <qhao@redhat.com>

Author: haoqing0110

kueue-addon/README.md

@@ -1,19 +1,29 @@
 # Kueue Addon for Open Cluster Management
 
-This addon provides an integration between Kueue and Open Cluster Management (OCM) to simplify MultiKueue setup and enhance multicluster scheduling capabilities.
+This addon provides an integration between [Kueue](https://kueue.sigs.k8s.io/) and Open Cluster Management (OCM) to simplify MultiKueue setup and enhance multicluster scheduling capabilities.
 
 This integration is designed for Kueue users who want to leverage OCM's capabilities to improve their multicluster workload scheduling experience. 
 
-The Kueue addon offers two primary benefits:
+The Kueue addon offers below benefits:
 
 - **Simplified MultiKueue Setup**
    - Automates the generation of MultiKueue specific Kubeconfig.
    - Streamlines the configuration of MultiKueue resources.
+   - Eliminates manual secret management.
+
+- **Centralizing Resource management**
+   - Manage spoke resources (ResourceFlavor, ClusterQueue, LocalQueue) from a single hub.
+   - Template-based deployment.
 
 - **Enhanced Multicluster Scheduling**
    - Integrates with OCM's placement with MultiKueue by implementing an admission check controller.
    - Generate MultiKueueConfig & MultiKueueCluster dynamically based on OCM placement decision.
+   - Supports advanced placement strategies.
 
+- **Flexible Installation Options**
+   - Standard installation for existing Kueue setups.
+   - Operator-based installation for OpenShift/OLM environments.
+   - Cluster proxy support for enhanced connectivity.
 
 ## Description
 
@@ -109,45 +119,128 @@ This controller is running on the hub, contains a credential controller and an a
 ### Addon chart
 - **Addon deployment:** Deploy [Kueue addon controllers](#kueue-addon-controller) on the hub.
 - **Addon Template:** To deploy `ResourceFlavor`, `ClusterQueue` and `LocalQueue` resources need by MultiKueue to spoke clusters.
-- **Other addon files:**  `ClusterManagementAddOn`, `ClusterRole`, `ClusterRoleBinding`, `ManagedClusterSetBinding`, `Placement` etc. 
+- **Kueue Operator Template:** Optional operator-based Kueue installation on spoke clusters.
+- **Other addon files:**  `ClusterManagementAddOn`, `ClusterRole`, `ClusterRoleBinding`, `ManagedClusterSetBinding`, `Placement` etc.
 
 ## Prerequisites
 
 - Open Cluster Management (OCM) installed with the following addons:
   - [Cluster Permission Addon](https://github.com/open-cluster-management-io/cluster-permission)
   - [Managed Service Account Addon](https://github.com/open-cluster-management-io/managed-serviceaccount)
-- Kueue already installed on the hub and spoke cluster
-- MultiKueue enabled on the hub
+  - [Cluster Proxy Addon](https://github.com/open-cluster-management-io/cluster-proxy) (Optional) Enables hub-to-spoke connectivity for enhanced networking.
+- Kueue installed:
+  - Hub Cluster with [Kueue](https://kueue.sigs.k8s.io/docs/installation/) installed and MultiKueue enabled.
+  - Spoke Clusters with [Kueue](https://kueue.sigs.k8s.io/docs/installation/) pre-installed, or let this addon install Kueue via [operator](https://github.com/openshift/kueue-operator) (OpenShift/OLM environments).
 
 ## Quick Start
 
-**_Important_**: The addon requires you already installed OCM, Cluster Permission Addon, Managed Service Account Addon and Kueue. 
-The whole setup steps about this solution, please refer to this [Kueue Integration Solution](https://github.com/open-cluster-management-io/ocm/blob/main/solutions/kueue-admission-check).
-
-## Installation
-
-On the hub cluster, deploy the addon.
+For a complete setup including all prerequisites on Kind:
 
 ```bash
-make deploy
+./build/setup-env.sh
 ```
 
-You can install the addons via the helm charts.
+## Installation
+
+### Step 1: Add Helm Repository
 
 ```bash
 $ helm repo add ocm https://open-cluster-management.io/helm-charts/
 $ helm repo update
 $ helm search repo ocm/kueue-addon
 NAME            CHART VERSION   APP VERSION     DESCRIPTION
 ocm/kueue-addon <chart-version> <app-version>           A Helm chart for Open Cluster Management Kueue ...
+```
+
+### Step 2: Choose Your Installation Method
+
+#### Option A: Standard Installation (Kueue Pre-installed)
+
+For environments where Kueue is already installed:
+
+```bash
 $ helm install \
     -n open-cluster-management-addon --create-namespace \
     kueue-addon ocm/kueue-addon
     # Uncomment the following lines to customize your installation:
-    # --set skipClusterSetBinding=true \
-    # --set image.tag=<chart-version> \
+    # --set skipClusterSetBinding=true
+```
+
+#### Option B: Operator-based Installation (OpenShift/OLM)
+
+In operator-based environments (for example, OpenShift with OLM), the addon can install the [Kueue operator](https://github.com/openshift/kueue-operator) for you.
+
+Prepare a values.operator.yaml with below content:
+
+```yaml
+kueue:
+  namespace: "openshift-kueue-operator"
+
+# Install Kueue via Operator
+installKueueViaOperator: true
+
+# Operator Lifecycle Manager RBAC configuration
+operatorLifecycleManager:
+  clusterRoleBindingName: kueue-operator-lifecycle-manager-rolebinding
+  clusterRoleName: system:controller:operator-lifecycle-manager
+
+# Kueue Operator configuration
+kueueOperator:
+  name: kueue-operator
+  namespace: openshift-kueue-operator
+  operatorGroupName: openshift-kueue-operator
+  channel: stable-v1.0
+  source: redhat-operators
+  sourceNamespace: openshift-marketplace
+  startingCSV: kueue-operator.v1.0.1
+
+# Cert Manager Operator configuration
+certManagerOperator:
+  name: openshift-cert-manager-operator
+  namespace: cert-manager-operator
+  operatorGroupName: cert-manager-operator
+  channel: stable-v1
+  source: redhat-operators
+  sourceNamespace: openshift-marketplace
+  startingCSV: cert-manager-operator.v1.17.0
+
+# Cluster proxy configuration
+clusterProxy:
+  url: "https://<cluster-proxy-url>"
+
+networkPolicy:
+  name: kueue-allow-egress-cluster-proxy-dns
+  namespace: "openshift-kueue-operator"
+  spec:
+    podSelector:
+      matchLabels:
+        app.openshift.io/name: kueue
+    policyTypes:
+    - Egress
+    egress:
+    - ports:
+      - port: 80
+        protocol: TCP
+    - ports:
+      - port: 443
+        protocol: TCP
+    - to:
+      - namespaceSelector:
+          matchLabels:
+            kubernetes.io/metadata.name: <cluster-proxy-namespace>
+```
+
+```bash
+$ helm install \
+    -n open-cluster-management-addon --create-namespace \
+    kueue-addon ocm/kueue-addon \
+    -f values.operator.yaml
+    # Uncomment the following lines to customize your installation:
+    # --set skipClusterSetBinding=true
 ```
 
+## Verification
+
 To confirm the installation from hub:
 
 ```bash
@@ -182,7 +275,7 @@ default-flavor   4h28m
 
 ## Usage
 
-The usage please refer to this [Kueue Integration Solution](https://github.com/open-cluster-management-io/ocm/blob/main/solutions/kueue-admission-check).
+For detailed usage examples and advanced scenarios, refer to the [Kueue Integration Solution](https://github.com/open-cluster-management-io/ocm/blob/main/solutions/kueue-admission-check).
 
 ## Design Details and Workflow
 
@@ -293,4 +386,3 @@ This automation greatly reduces manual effort and ensures that MultiKueue enviro
 - The admission check controller watches for `AdmissionCheck` resources referencing OCM `Placement`.
 - The admission check controller watches the `PlacementDecision`, creates or updates `MultiKueueCluster` resources with the kubeconfig details for each cluster, and also update these clusters in the `MultiKueueConfig` resource.
 - Finally, admission check controller updates the `AdmissionCheck` condition to true, indicating successful generation of the `MultiKueueConfig` and `MultiKueueCluster`, readying the [MultiKueue](https://kueue.sigs.k8s.io/docs/concepts/multikueue/) environment for job scheduling.
-

kueue-addon/build/setup-env.sh

@@ -123,6 +123,13 @@ install_ocm_addons() {
      --set enableAddOnDeploymentConfig=true \
      --set hubDeployMode=AddOnTemplate
 
+  echo "Install cluster-proxy"
+  helm upgrade --install \
+    -n open-cluster-management-addon --create-namespace \
+    cluster-proxy ocm/cluster-proxy \
+    --set installByPlacement.placementName=global \
+    --set installByPlacement.placementNamespace=open-cluster-management-addon
+
   echo "Install cluster-permission"
   helm upgrade --install \
     -n open-cluster-management --create-namespace \

kueue-addon/charts/kueue-addon/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: kueue-addon
 description: A Helm chart for Open Cluster Management Kueue Addon
 type: application
-version: 0.1.3
+version: 0.1.4
 appVersion: "0.11.9"
 maintainers:
   - name: haoqing0110

kueue-addon/charts/kueue-addon/templates/_operator.tpl

@@ -0,0 +1,75 @@
+{{/*
+Operator Resources - Single Template
+*/}}
+{{- define "kueue-addon.operatorResources" -}}
+{{- if and .Values.kueueOperator .Values.certManagerOperator .Values.operatorLifecycleManager }}
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: ClusterRoleBinding
+  metadata:
+    name: {{ .Values.operatorLifecycleManager.clusterRoleBindingName }}
+  roleRef:
+    apiGroup: rbac.authorization.k8s.io
+    kind: ClusterRole
+    name: {{ .Values.operatorLifecycleManager.clusterRoleName }}
+  subjects:
+  - kind: ServiceAccount
+    name: klusterlet-work-sa
+    namespace: open-cluster-management-agent
+- apiVersion: v1
+  kind: Namespace
+  metadata:
+    name: {{ .Values.kueueOperator.namespace }}
+- apiVersion: operators.coreos.com/v1
+  kind: OperatorGroup
+  metadata:
+    name: {{ .Values.kueueOperator.operatorGroupName }}
+    namespace: {{ .Values.kueueOperator.namespace }}
+  spec:
+    upgradeStrategy: Default
+- apiVersion: operators.coreos.com/v1alpha1
+  kind: Subscription
+  metadata:
+    name: {{ .Values.kueueOperator.name }}
+    namespace: {{ .Values.kueueOperator.namespace }}
+  spec:
+    channel: {{ .Values.kueueOperator.channel }}
+    installPlanApproval: Automatic
+    name: {{ .Values.kueueOperator.name }}
+    source: {{ .Values.kueueOperator.source }}
+    sourceNamespace: {{ .Values.kueueOperator.sourceNamespace }}
+    startingCSV: {{ .Values.kueueOperator.startingCSV }}
+- apiVersion: kueue.openshift.io/v1
+  kind: Kueue
+  metadata:
+    name: cluster
+    labels:
+      app.kubernetes.io/name: kueue-operator
+  spec:
+    {{- if .Values.kueueCR.spec }}
+{{ toYaml .Values.kueueCR.spec | indent 4 }}
+    {{- end }}
+- apiVersion: v1
+  kind: Namespace
+  metadata:
+    name: {{ .Values.certManagerOperator.namespace }}
+- apiVersion: operators.coreos.com/v1
+  kind: OperatorGroup
+  metadata:
+    name: {{ .Values.certManagerOperator.operatorGroupName }}
+    namespace: {{ .Values.certManagerOperator.namespace }}
+  spec:
+    upgradeStrategy: Default
+- apiVersion: operators.coreos.com/v1alpha1
+  kind: Subscription
+  metadata:
+    name: {{ .Values.certManagerOperator.name }}
+    namespace: {{ .Values.certManagerOperator.namespace }}
+  spec:
+    channel: {{ .Values.certManagerOperator.channel }}
+    installPlanApproval: Automatic
+    name: {{ .Values.certManagerOperator.name }}
+    source: {{ .Values.certManagerOperator.source }}
+    sourceNamespace: {{ .Values.certManagerOperator.sourceNamespace }}
+    startingCSV: {{ .Values.certManagerOperator.startingCSV }}
+{{- end }}
+{{- end }}

kueue-addon/charts/kueue-addon/templates/addon-template.yaml

@@ -62,3 +62,7 @@ spec:
             name: {{ .Values.localQueue.name}}
           spec:
             clusterQueue: {{ .Values.clusterQueue.name}}
+{{- if .Values.installKueueViaOperator }}
+{{- include "kueue-addon.operatorResources" . | nindent 8 }}
+{{- end }}
+

kueue-addon/charts/kueue-addon/templates/deployment.yaml

@@ -21,6 +21,13 @@ spec:
           args:
             - "/kueue-addon-controller"
             - "hub"
+          env:
+            - name: KUEUE_NAMESPACE
+              value: {{ .Values.kueue.namespace }}
+            {{- if .Values.clusterProxy.url }}
+            - name: CLUSTER_PROXY_URL
+              value: {{ .Values.clusterProxy.url }}
+            {{- end }}
 
 ---
 

kueue-addon/charts/kueue-addon/templates/networkpolicy.yaml

@@ -0,0 +1,9 @@
+{{- if .Values.networkPolicy }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Values.networkPolicy.name }}
+  namespace: {{ .Values.networkPolicy.namespace }}
+spec:
+{{ toYaml .Values.networkPolicy.spec | indent 2 }}
+{{- end }}

kueue-addon/charts/kueue-addon/templates/placement.yaml

@@ -1,4 +1,4 @@
-{{ if not .Values.skipClusterSetBinding }}
+{{- if not .Values.skipClusterSetBinding }}
 apiVersion: cluster.open-cluster-management.io/v1beta2
 kind: ManagedClusterSetBinding
 metadata:

kueue-addon/charts/kueue-addon/values.yaml

@@ -4,7 +4,7 @@
 # Image configuration
 image:
   repository: quay.io/open-cluster-management/kueue-addon
-  tag:
+  tag: ""  # defaults to chart appVersion
   pullPolicy: IfNotPresent
 
 # Skip cluster set binding
@@ -38,3 +38,59 @@ clusterQueue:
 localQueue:
   name: user-queue
   namespace: default
+
+# This is the namespace where the kueue controller is installed, will copy the multikueue secret to this namespace
+# Default is kueue-system
+kueue:
+  namespace: "kueue-system"
+
+# Cluster proxy configuration (optional)
+clusterProxy:
+  # URL for the cluster proxy service, used to access managed clusters through a proxy
+  # When set, the controller will use this URL to connect to managed clusters
+  # Example: "https://cluster-proxy.example.com/clusters/"
+  url: ""
+
+# NetworkPolicy configuration (uncomment when installKueueViaOperator enabled)
+# Uncomment when using operator-based installation with network restrictions
+# networkPolicy:
+#   name: kueue-allow-egress-cluster-proxy-dns
+#   namespace: "kueue-system"
+#   spec: {}
+
+# Enable Kueue installation via operator
+installKueueViaOperator: false
+
+# Operator Lifecycle Manager RBAC configuration (uncomment when installKueueViaOperator enabled)
+# operatorLifecycleManager:
+#   clusterRoleBindingName: <cluster-role-binding-name>
+#   clusterRoleName: <cluster-role-name>
+
+# Kueue Operator configuration (uncomment when installKueueViaOperator enabled)
+# kueueOperator:
+#   name: <operator-name>
+#   namespace: <operator-namespace>
+#   operatorGroupName: <operator-group-name>
+#   channel: <channel-version>
+#   source: <operator-source>
+#   sourceNamespace: <source-namespace>
+#   startingCSV: <starting-csv>
+
+# Cert Manager Operator configuration (uncomment when installKueueViaOperator enabled)
+# certManagerOperator:
+#   name: <cert-manager-operator-name>
+#   namespace: <cert-manager-namespace>
+#   operatorGroupName: <cert-manager-operator-group>
+#   channel: <cert-manager-channel>
+#   source: <cert-manager-source>
+#   sourceNamespace: <cert-manager-source-namespace>
+#   startingCSV: <cert-manager-starting-csv>
+
+# Kueue CR customization (uncomment when installKueueViaOperator enabled)
+# kueueCR:
+#   spec:
+#     config:
+#       integrations:
+#         frameworks:
+#         - BatchJob
+#     managementState: Managed