Skip to content

Commit a6e47e6

Browse files
author
Klavs Klavsen
committed
Merge pull request 'Document kube-prometheus + OpenObserve monitoring model with OTel log ingestion and alerting' (#1686) from docs/openobserve-monitoring-guide into master
Reviewed-on: https://gitea.obmondo.com/EnableIT/KubeAid/pulls/1686
2 parents 88e7120 + 126f53a commit a6e47e6

8 files changed

Lines changed: 213 additions & 3 deletions

File tree

‎argocd-helm-charts/graylog/README.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,25 @@
11
# Graylog docs
22

3-
## Add the graylog username and password into sealed secret
3+
## Log monitoring in KubeAid
4+
5+
Graylog is a **log-only** monitoring option in KubeAid. It runs alongside
6+
[`kube-prometheus`](../../docs/kubeaid/prometheus-configuration.md) (Prometheus, Alertmanager, Grafana), which
7+
continues to handle metrics and metric-based alerts.
8+
9+
| | |
10+
| - | - |
11+
| **Scope** | Log ingestion, search, pipelines, and log-based alerting |
12+
| **Log collection** | Fluent Bit, Fluentd, Beats, Syslog, GELF, and other Graylog inputs |
13+
| **Prometheus integration** | None — Graylog does not pull metrics or alerts from Prometheus |
14+
| **Storage** | OpenSearch (indexed logs) and MongoDB (Graylog metadata) |
15+
16+
See [Monitoring](../../docs/monitoring.md) for how Graylog compares to
17+
OpenObserve and OpenSearch + Kibana.
18+
19+
The sections below cover installation, configuration, and operations.
20+
21+
---
22+
423

524
```sh
625
# pwgen 20 1 | tr -d '\n' > graylog-password

‎argocd-helm-charts/openobserve/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,22 @@
11
# OpenObserve Installation Guide
22

3+
## Log monitoring in KubeAid
4+
5+
OpenObserve extends the KubeAid monitoring stack with log ingestion, search, and alerting. It uses the
6+
**OpenTelemetry** standard and integrates with
7+
[`kube-prometheus`](../../docs/kubeaid/prometheus-configuration.md).
8+
9+
| | |
10+
| - | - |
11+
| **Scope** | Logs; can also pull metrics and alerts from Prometheus |
12+
| **Log collection** | OpenTelemetry Collector (`OTLP`) — deploy the collector before OpenObserve |
13+
| **Prometheus integration** | Pulls metrics and alerts from Prometheus; metric alerting still routed via Alertmanager |
14+
| **Alerting** | Log-based alerts in OpenObserve alongside Prometheus metric alerts |
15+
16+
See [Monitoring](../../docs/monitoring.md) for how OpenObserve compares to
17+
Graylog and OpenSearch + Kibana. For collector architecture and receivers, see the
18+
[OpenObserve Collector reference](./charts/openobserve-collector/docs/README.md).
19+
320
> **Tip:** Before installing OpenObserve, you must deploy the **OpenTelemetry Collector**.
421
> Add it as a regular Argo CD application in your `kubeaid-config` repository.
522

‎argocd-helm-charts/openobserve/charts/openobserve-collector/docs/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
# OpenObserve Collector — Data Collection Reference
22

3+
## Role in KubeAid monitoring
4+
5+
The OpenObserve Collector is the **OpenTelemetry ingestion layer** for OpenObserve in KubeAid. It collects logs,
6+
metrics, traces, and Kubernetes object state from the cluster and ships them to OpenObserve over OTLP.
7+
8+
It also scrapes Prometheus-compatible targets (cadvisor, kube-state-metrics, CoreDNS, annotated pods, etc.), which is
9+
how OpenObserve pulls cluster metrics alongside logs. Metric alerting in Prometheus/Alertmanager is unchanged — see
10+
[Monitoring](../../../../docs/monitoring.md).
11+
12+
For OpenObserve installation and configuration, see the [OpenObserve chart README](../../README.md).
13+
14+
---
15+
316
This chart deploys an OpenTelemetry Collector stack (agent DaemonSet + gateway StatefulSet) that collects logs, metrics, traces, and Kubernetes object state from your cluster and ships them to OpenObserve.
417

518
## Architecture

‎argocd-helm-charts/opensearch-dashboards/charts/opensearch-dashboards/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,27 @@
11
# OpenSearch Dashboards Helm Chart
22

3+
## Log monitoring in KubeAid
4+
5+
[OpenSearch Dashboards](https://github.com/opensearch-project/OpenSearch-Dashboards) is the Kibana-compatible UI for
6+
querying and visualizing logs stored in
7+
[OpenSearch](../../opensearch/README.md). Together they form an ELK-style log monitoring stack — an alternative to
8+
Graylog.
9+
10+
| | |
11+
| - | - |
12+
| **Scope** | Log search and dashboards (logs only) |
13+
| **Requires** | An OpenSearch cluster |
14+
| **Prometheus integration** | None — metrics stay in [`kube-prometheus`](../../../../docs/kubeaid/prometheus-configuration.md) |
15+
| **Kibana (Elastic)** | Not packaged in KubeAid; OpenSearch Dashboards is the supported equivalent |
16+
17+
**Deployment status:** KubeAid includes this chart, but OpenSearch Dashboards has not been deployed as a standalone log
18+
UI stack to date. OpenSearch is commonly deployed as Graylog's storage backend instead.
19+
20+
See [Monitoring](../../../../docs/monitoring.md) for how this compares to
21+
OpenObserve and Graylog.
22+
23+
---
24+
325
This Helm chart installs [OpenSearch Dashboards](https://github.com/opensearch-project/OpenSearch-Dashboards) with configurable TLS, RBAC and much more configurations. This chart caters to a number of different use cases and setups.
426

527
- [Requirements](#requirements)

‎argocd-helm-charts/opensearch/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,31 @@
11
# Opensearch Cluster
22

3+
## Log monitoring in KubeAid
4+
5+
OpenSearch is a log storage and search engine in KubeAid. It is used in two ways:
6+
7+
1. **As Graylog's backend** — stores indexed logs written by Graylog (common deployment pattern)
8+
2. **As a standalone ELK-style stack** — paired with
9+
[OpenSearch Dashboards](../opensearch-dashboards/charts/opensearch-dashboards/README.md) for direct log search
10+
and visualization; an alternative to Graylog
11+
12+
In both cases, [`kube-prometheus`](../../docs/kubeaid/prometheus-configuration.md) remains responsible for metrics and
13+
metric-based alerts. OpenSearch handles logs only.
14+
15+
| | |
16+
| - | - |
17+
| **Scope** | Log storage, search, and (with Dashboards) visualization |
18+
| **Log collection** | Fluent Bit, Fluentd, OpenTelemetry Collector, or via Graylog |
19+
| **Prometheus integration** | None for logs; cluster health metrics can be scraped via ServiceMonitor |
20+
| **Kibana** | Not packaged in KubeAid — use the OpenSearch Dashboards chart instead |
21+
22+
See [Monitoring](../../docs/monitoring.md) for a comparison of all log
23+
monitoring options. When used with Graylog, see also the [Graylog chart docs](../graylog/README.md).
24+
25+
The sections below cover cluster operations.
26+
27+
---
28+
329
## How to increase the ES index limit
430

531
### Sample issue which you see in graylog logs

‎docs/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ Guides for ongoing cluster management:
5656

5757
| Guide | Description |
5858
| ------- | ------------- |
59+
| [Monitoring](./monitoring.md) | kube-prometheus metrics stack and log monitoring options (OpenObserve, Graylog, OpenSearch + Kibana) |
5960
| [Pod Autoscaling](./operations/monitoring/pod-autoscaling.md) | HPA and VPA configuration |
6061
| [Prometheus Namespaces](./operations/monitoring/prometheus-namespaces.md) | Namespace-level monitoring |
6162

@@ -83,6 +84,7 @@ Guides for ongoing cluster management:
8384
| [Features Technical Details](./kubeaid/features-technical-details.md) | In-depth feature documentation |
8485
| [Helm Umbrella Pattern](./kubeaid/helm-umbrella-pattern.md) | How KubeAid manages applications |
8586
| [Prometheus Configuration](./kubeaid/prometheus-configuration.md) | Configuring monitoring with kube-prometheus |
87+
| [Monitoring](./monitoring.md) | Metrics and log monitoring overview |
8688
| [GitOps Drift Detection](./kubeaid/gitops-drift-detection.md) | ArgoCD sync status and alerting |
8789

8890
## Support

‎docs/kubeaid/prometheus-configuration.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,8 @@ kubectl exec -it $GrafanaPod -n monitoring -- grafana-cli admin reset-admin-pass
271271

272272
## See Also
273273

274+
- [Monitoring](../monitoring.md)
274275
- [kube-prometheus Build Documentation](../../build/kube-prometheus/README.md)
275-
- [Pod Autoscaling Guide](../operations/monitoring/pod-autoscaling.md)
276-
- [Prometheus Namespace Configuration](../operations/monitoring/prometheus-namespaces.md)
276+
- [Pod Autoscaling Guide](./operations/monitoring/pod-autoscaling.md)
277+
- [Prometheus Namespace Configuration](./operations/monitoring/prometheus-namespaces.md)
277278
- [Upstream kube-prometheus](https://github.com/prometheus-operator/kube-prometheus)

‎docs/monitoring.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Monitoring in KubeAid
2+
3+
KubeAid monitoring has two layers:
4+
5+
1. **Metrics** — `kube-prometheus` (Prometheus, Alertmanager, Grafana)
6+
2. **Logs** — one of OpenObserve, Graylog, or OpenSearch + Kibana
7+
8+
Per-stack setup and operations are documented in each application's Helm chart README (linked below).
9+
10+
## Metrics: kube-prometheus
11+
12+
[kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) is the default monitoring stack. It provides:
13+
14+
- **Prometheus** — scrapes metrics from ServiceMonitors, PodMonitors, and exporters across the cluster
15+
- **Alertmanager** — routes metric-based alerts to notification channels
16+
- **Grafana** — dashboards for metrics visualization
17+
18+
Configuration is managed per cluster via Jsonnet (`<cluster-name>-vars.jsonnet`) and built into Kubernetes manifests.
19+
See [Prometheus Configuration](./kubeaid/prometheus-configuration.md) for details.
20+
21+
```mermaid
22+
flowchart LR
23+
subgraph Cluster["Kubernetes Cluster"]
24+
Apps["Apps and System Components"]
25+
Exporters["Prometheus Exporters / ServiceMonitors"]
26+
end
27+
28+
subgraph Metrics["kube-prometheus"]
29+
Prom["Prometheus"]
30+
AM["Alertmanager"]
31+
Graf["Grafana"]
32+
end
33+
34+
Apps --> Exporters
35+
Exporters --> Prom
36+
Prom --> AM
37+
Prom --> Graf
38+
```
39+
40+
## Log monitoring
41+
42+
Log monitoring runs alongside `kube-prometheus`. Each option handles log ingestion, search, and log-based alerting on
43+
its own — none of them replace Prometheus for metrics.
44+
45+
| Option | Scope | Log collection | Prometheus integration | Application docs |
46+
| ------ | ----- | -------------- | ---------------------- | ---------------- |
47+
| OpenObserve | Logs, metrics, and alerts | OpenTelemetry (`OTLP`) | Pulls metrics and alerts from Prometheus; alerts on logs | [openobserve](../argocd-helm-charts/openobserve/README.md) |
48+
| Graylog | Logs only | Fluent Bit, Fluentd, Beats, Syslog, GELF, etc. | None (metrics stay in kube-prometheus) | [graylog](../argocd-helm-charts/graylog/README.md) |
49+
| OpenSearch + Kibana | Logs only (ELK-style) | Fluent Bit, Fluentd, OpenTelemetry Collector, etc. | None (metrics stay in kube-prometheus) | [opensearch](../argocd-helm-charts/opensearch/README.md), [opensearch-dashboards](../argocd-helm-charts/opensearch-dashboards/charts/opensearch-dashboards/README.md) |
50+
51+
### OpenObserve
52+
53+
OpenObserve uses the **OpenTelemetry** standard for ingestion. It can ingest logs via OpenTelemetry Collector, pull
54+
metrics and alerts from Prometheus, and provide log-based search and alerting.
55+
56+
- [OpenObserve Helm chart](../argocd-helm-charts/openobserve/README.md)
57+
- [OpenObserve Collector reference](../argocd-helm-charts/openobserve/charts/openobserve-collector/docs/README.md)
58+
59+
### Graylog
60+
61+
Graylog focuses on **log collection and management**. Logs are shipped using agents such as Fluent Bit or other
62+
supported inputs. Graylog does not integrate with Prometheus for metrics or alerting.
63+
64+
- [Graylog Helm chart](../argocd-helm-charts/graylog/README.md)
65+
66+
### OpenSearch and Kibana
67+
68+
OpenSearch with Kibana (or OpenSearch Dashboards) is an alternative to Graylog: centralized log storage, search, and
69+
visualization without a Graylog management layer. KubeAid includes Helm charts for OpenSearch and OpenSearch
70+
Dashboards; Kibana itself is not packaged.
71+
72+
- [OpenSearch Helm chart](../argocd-helm-charts/opensearch/README.md)
73+
- [OpenSearch Dashboards Helm chart](../argocd-helm-charts/opensearch-dashboards/charts/opensearch-dashboards/README.md)
74+
75+
```mermaid
76+
flowchart LR
77+
subgraph Sources["Log sources"]
78+
Apps["Applications"]
79+
Collectors["Fluent Bit / OTel Collector"]
80+
end
81+
82+
subgraph Logs["Log stack (one of)"]
83+
OO["OpenObserve"]
84+
GL["Graylog"]
85+
OSD["OpenSearch + Dashboards"]
86+
end
87+
88+
subgraph Metrics["kube-prometheus"]
89+
Prom["Prometheus"]
90+
end
91+
92+
Collectors --> OO
93+
Collectors --> GL
94+
Collectors --> OSD
95+
Apps --> Collectors
96+
Prom -. metrics and alerts .-> OO
97+
```
98+
99+
## Alerting strategy
100+
101+
- **Metric alerts** — Prometheus rules evaluated by Prometheus, routed by Alertmanager
102+
- **Log alerts** — evaluated by the log stack (OpenObserve, Graylog, or OpenSearch)
103+
104+
Metrics tell you that something is unhealthy; logs help explain why.
105+
106+
## Further reading
107+
108+
- [Prometheus Configuration](./kubeaid/prometheus-configuration.md) — kube-prometheus setup and Jsonnet build
109+
- [Prometheus Namespaces](./operations/monitoring/prometheus-namespaces.md) — namespace scrape scope
110+
- [Pod Autoscaling](./operations/monitoring/pod-autoscaling.md) — HPA with custom metrics from Prometheus

0 commit comments

Comments
 (0)