✨ Enable federated learning observability (open-cluster-management-io#55)

* Obs-addon

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Add reference to otel addon in README

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Use markdown

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Fix: use the correct namespace of kepler.

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Update README

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Add doc: configure-environment-observability

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* Update document

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* modify dir name

Signed-off-by: mrrr61 <mrrr61@outlook.com>

* modify dir name: otel_addon to otel-addon

Signed-off-by: mrrr61 <mrrr61@outlook.com>

---------

Signed-off-by: mrrr61 <mrrr61@outlook.com>

Author: mrrr3d

federated-learning-controller/README.md

@@ -51,6 +51,10 @@ cluster1   true           https://cluster1-control-plane:6443   True     True
 cluster2   true           https://cluster2-control-plane:6443   True     True        3m
 ```
 
+#### Optional: Configure Environment for Observability
+
+Please refer to the [Observability Setup](docs/configure-environment-observability.md) documentation for more details.
+
 --- 
 
 ### Deploy Federated Learning Controller

federated-learning-controller/assets/images/obs_architecture.png

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# Generate Root CA
+openssl genrsa -out root-ca.key 4096
+openssl req -x509 -new -nodes -key root-ca.key -sha256 -days 3650 \
+  -subj "/CN=My Root CA" \
+  -out root-ca.crt
+
+# Generate Client CA
+openssl genrsa -out client-ca.key 4096
+openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3650 \
+  -subj "/CN=My Client CA" \
+  -out client-ca.crt
+
+# Generate Server Key + CSR + Signed Cert
+openssl genrsa -out server.key 2048
+openssl req -new -key server.key -out server.csr -config san.cnf
+
+# Sign server cert with Root CA
+openssl x509 -req -in server.csr -CA root-ca.crt -CAkey root-ca.key \
+  -CAcreateserial -out server.crt -days 825 -sha256 -extensions v3_req -extfile san.cnf
+
+# Remove the secrets if they exist
+kubectl --context kind-hub delete secret prometheus-tls -n monitoring --ignore-not-found
+kubectl --context kind-hub delete secret otel-signer -n open-cluster-management-hub --ignore-not-found
+
+# create prometheus-tls secret in monitoring namespace
+kubectl --context kind-hub create secret generic prometheus-tls -n monitoring \
+  --from-file=server.crt=server.crt\
+  --from-file=server.key=server.key \
+  --from-file=client-ca.crt=client-ca.crt
+
+# create otel-signer secret in open-cluster-management-hub namespace
+kubectl --context kind-hub create secret tls otel-signer -n open-cluster-management-hub \
+  --cert=client-ca.crt\
+  --key=client-ca.key
+
+# replace root-ca.crt in deploy/resources/addon-template.yaml
+awk '{print "              " $0}' root-ca.crt > root-ca.crt.tmp
+sed -i "/PROM_WEB_ROOT_CA/{
+        r root-ca.crt.tmp
+        d
+    }" ../../deploy/resources/addon-template.yaml
+rm root-ca.crt.tmp

federated-learning-controller/deploy/obs/hack/certs/generate-certs.sh

@@ -0,0 +1,13 @@
+[ req ]
+distinguished_name = req_distinguished_name
+req_extensions = v3_req
+prompt = no
+
+[ req_distinguished_name ]
+CN = hub-control-plane
+
+[ v3_req ]
+subjectAltName = @alt_names
+
+[ alt_names ]
+DNS.1 = hub-control-plane

federated-learning-controller/deploy/obs/hack/certs/san.cnf

@@ -0,0 +1,19 @@
+prometheus:
+  prometheusSpec:
+    enableRemoteWriteReceiver: true
+    web:
+      tlsConfig:
+        keySecret:
+          key: server.key
+          name: prometheus-tls
+        cert:
+          secret:
+            key: server.crt
+            name: prometheus-tls
+        client_ca:
+          secret:
+            key: client-ca.crt
+            name: prometheus-tls
+        clientAuthType: RequestClientCert
+  service:
+    type: NodePort

federated-learning-controller/deploy/obs/hack/prom/values.yaml

@@ -0,0 +1,7 @@
+resources:
+- resources/addon-template.yaml
+- resources/placement.yaml
+- resources/cluster-management-addon.yaml
+- resources/addon-deployment-config.yaml
+- resources/addonmanager.role.yaml
+- resources/addonmanager.rolebinding.yaml

federated-learning-controller/deploy/obs/otel-addon/kustomization.yaml

@@ -0,0 +1,9 @@
+apiVersion: addon.open-cluster-management.io/v1alpha1
+kind: AddOnDeploymentConfig
+metadata:
+  name: otel-config
+  namespace: open-cluster-management-hub
+spec:
+  customizedVariables:
+  - name: PROM_REMOTE_WRITE_ENDPOINT
+    value: https://hub-control-plane:30090/api/v1/write