Otel addon (open-cluster-management-io#47)

* initial commit for otel-addon

Signed-off-by: clyang82 <chuyang@redhat.com>

* update with placement, secrets, etc

Signed-off-by: clyang82 <chuyang@redhat.com>

* minor updates

Signed-off-by: clyang82 <chuyang@redhat.com>

* Address review comments

Signed-off-by: clyang82 <chuyang@redhat.com>

---------

Signed-off-by: clyang82 <chuyang@redhat.com>

Author: clyang82

open-telemetry-addon/.dockerignore

@@ -3,33 +3,110 @@
 ## What is OtelCollector-Addon?
 
 OtelCollector Addon is a pluggable addon working on the extensibility provided by [addon-framework](https://github.com/open-cluster-management-io/addon-framework)
-which automates the installation of otelCollector on both hub cluster and managed clusters and jaeget-all-in-one on hub cluster for processing and storing the traces.
+which automates the installation of otelCollector on the managed clusters.
 
-OtelCollecotr Addon consists of two components:
 
-- __Addon-Manager__: Manages the installation of hub components for setting up the Addon
+### Prerequisite
 
-- __Addon-Agent__: Manages the installation of collector agents in the managed clustres.
+The otel-addon depends on prometheus-stack installed on the hub cluster. Before you get started, you must have a OCM environment setuped. You can also follow our recommended [quick start guide](https://open-cluster-management.io/docs/getting-started/quick-start/) to set up a playgroud OCM environment.
 
-The overall architecture is shown below:
+#### Add Helm Repo
 
-![Arch](./hack/picture/arch.png)
+```shell
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+```
+
+#### Install Prometheus
+
+1. You need to create `monitoring` namespace in hub cluster with this command:
+```
+kubectl --context kind-hub create namespace monitoring
+```
+
+2. You have to run the script to generate the certs before installing prometheus with mTLS enabled. The script do the following things:
+- Generate root ca and key
+- Generate client ca and key
+- Generate server cert and key
+- Create prometheus-tls secret in monitoring namespace
+- Create otel-signer secret in open-cluster-management-hub namespace
 
+```shell
+cd hack/certs
+bash ./generate-certs.sh
+cd ../..
+```
 
-### Installing via Helm Chart :
+3. Install Prometheus with the following command:
 
 ```shell
-$ helm install \
-    -n open-cluster-management-addon --create-namespace \
-    otel-collector charts/otel
+helm --kube-context kind-hub install prometheus prometheus-community/kube-prometheus-stack \
+  --namespace monitoring \
+  -f ./hack/prom/values.yaml
 ```
 
-### Verifying the addOn :
+### Installing the otel-addon
+
+```shell
+kubectl --context kind-hub create namespace open-cluster-management-addon
+kubectl --context kind-hub apply -k deploy
 ```
-$ kubectl -n open-cluster-management-addon get pod
-NAME                                            READY   STATUS    RESTARTS   AGE
-jaeger-all-in-one-57dcf4b5-ltt46                1/1     Running   0          65s
-otel-collector-75fd748b4c-cbt7k                 1/1     Running   0          65s
-otel-collector-addon-manager-568885c78b-pl9db   1/1     Running   0          67s
+
+### Verifying the opentelemetry-collector in the managed clusters
+```
+kubectl --context kind-cluster1 get pod -n open-cluster-management-agent-addon
+NAME                                       READY   STATUS    RESTARTS   AGE
+opentelemetry-collector-66fcdc8779-zvhff   1/1     Running   0          27s
 ```
 
+### Verify container advisor metrics
+```
+curl -k https://hub-control-plane:30090/api/v1/query?query=container_fs_inodes_total | jq .
+{
+  "status": "success",
+  "data": {
+    "resultType": "vector",
+    "result": [
+      {
+        "metric": {
+          "__name__": "container_fs_inodes_total",
+          "beta_kubernetes_io_arch": "amd64",
+          "beta_kubernetes_io_os": "linux",
+          "cluster_name": "cluster1",
+          "device": "overlay_0-157",
+          "id": "/",
+          "instance": "cluster1-control-plane",
+          "job": "opentelemetry-collector",
+          "kubernetes_io_arch": "amd64",
+          "kubernetes_io_hostname": "cluster1-control-plane",
+          "kubernetes_io_os": "linux"
+        },
+        "value": [
+          1751596235.174,
+          "104840504"
+        ]
+      },
+      {
+        "metric": {
+          "__name__": "container_fs_inodes_total",
+          "beta_kubernetes_io_arch": "amd64",
+          "beta_kubernetes_io_os": "linux",
+          "cluster_name": "cluster2",
+          "device": "overlay_0-313",
+          "id": "/",
+          "instance": "cluster2-control-plane",
+          "job": "opentelemetry-collector",
+          "kubernetes_io_arch": "amd64",
+          "kubernetes_io_hostname": "cluster2-control-plane",
+          "kubernetes_io_os": "linux"
+        },
+        "value": [
+          1751596235.174,
+          "104840504"
+        ]
+      }
+      ...
+    ]
+  }
+}
+```