Rewrite the monitoring doc for the current observability stack - #76
Draft
ashish1099 wants to merge 1 commit into
Draft
Rewrite the monitoring doc for the current observability stack#76ashish1099 wants to merge 1 commit into
ashish1099 wants to merge 1 commit into
Conversation
ashish1099
force-pushed
the
docs/observability
branch
from
July 12, 2026 15:05
21e2ac7 to
cebe743
Compare
The doc still described OpenObserve as pulling metrics and alerts from Prometheus, had no traces section, and predated the runtime-security charts. Document the three layers and their single owners: kube-prometheus for metrics and alerting, OpenTelemetry -> OpenObserve for logs and traces, and Tetragon/KubeArmor for runtime security events. Cover the agent/gateway split, auto-instrumentation for traces, the servicegraph service map, why the collector does not collect metrics, OpenObserve licensing against the Apache-2.0 OpenSearch alternative, the deprecated filebeat and loki-stack charts, and the rule that vendored subcharts are configured from the wrapper values.
ashish1099
force-pushed
the
docs/observability
branch
from
July 12, 2026 15:42
cebe743 to
5375fa4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites
docs/monitoring.mdto describe the observability stack as it actually is, and to record the reasoning behind it so the same questions don't get re-litigated.Three layers, one owner each
What this corrects
There is no single pane of glass. The previous text claimed Grafana reads logs and traces from an OpenObserve datasource. It does not —
build/kube-prometheus/common-template.jsonnetdefines a Prometheus datasource and nothing else, andgrafana_pluginsis empty. The community OpenObserve plugin is unsigned and absent from the Grafana catalog, so loading it would requireallow_loading_unsigned_plugins.The doc now states the trade-off plainly: Grafana for metrics and alerts, OpenObserve's own UI for logs and traces, pivoting between them by trace ID or timestamp. In exchange, no data is duplicated and each signal stays in the store that is good at it. Loki + Tempo is named as the only route to a single Grafana pane, for anyone who needs one.
What this adds
k8sobjectswould otherwise duplicate every event per node, andservicegraphneeds both sides of a call to draw an edge across nodes.All 16 relative links verified.