Skip to content

Rewrite the monitoring doc for the current observability stack - #76

Draft
ashish1099 wants to merge 1 commit into
masterfrom
docs/observability
Draft

Rewrite the monitoring doc for the current observability stack#76
ashish1099 wants to merge 1 commit into
masterfrom
docs/observability

Conversation

@ashish1099

@ashish1099 ashish1099 commented Jul 12, 2026

Copy link
Copy Markdown
Member

Rewrites docs/monitoring.md to describe the observability stack as it actually is, and to record the reasoning behind it so the same questions don't get re-litigated.

Three layers, one owner each

Layer Signal Owner
Metrics + alerting metrics, alerts kube-prometheus
Logs + traces logs, traces, k8s events OTel Collector → OpenObserve
Runtime security process/file/network events Tetragon or KubeArmor → OpenObserve

What this corrects

There is no single pane of glass. The previous text claimed Grafana reads logs and traces from an OpenObserve datasource. It does not — build/kube-prometheus/common-template.jsonnet defines a Prometheus datasource and nothing else, and grafana_plugins is empty. The community OpenObserve plugin is unsigned and absent from the Grafana catalog, so loading it would require allow_loading_unsigned_plugins.

The doc now states the trade-off plainly: Grafana for metrics and alerts, OpenObserve's own UI for logs and traces, pivoting between them by trace ID or timestamp. In exchange, no data is duplicated and each signal stays in the store that is good at it. Loki + Tempo is named as the only route to a single Grafana pane, for anyone who needs one.

What this adds

  • Why metrics are not moved to the log backend. ~2,170 alert definitions and ~350 dashboards come free from the vendored Jsonnet mixins; ~70 charts ship a ServiceMonitor or PodMonitor that would go inert without prometheus-operator; Alertmanager and prometheus-adapter have no equivalent. Prometheus is also the most portable component in the stack.
  • Why there are two collectors. The agent must be a DaemonSet because log files are node-local; the gateway must be central because k8sobjects would otherwise duplicate every event per node, and servicegraph needs both sides of a call to draw an edge across nodes.
  • Servicegraph's rationale, stated rather than assumed: it is derived from traces, has no scrape endpoint, and is viewed alongside the traces it was built from.
  • OpenObserve licensing as a real comparison, documenting the OSS + traefik-forward-auth route for an auditable stack — including its honest limits (forward-auth authenticates at the ingress, not inside OpenObserve; AGPL is copyleft).
  • Alerting: log-backend alerts should be routed into Alertmanager rather than wired straight to a notification channel, so it stays the single path to on-call.

All 16 relative links verified.

@ashish1099
ashish1099 force-pushed the docs/observability branch from 21e2ac7 to cebe743 Compare July 12, 2026 15:05
The doc still described OpenObserve as pulling metrics and alerts from
Prometheus, had no traces section, and predated the runtime-security charts.

Document the three layers and their single owners: kube-prometheus for metrics
and alerting, OpenTelemetry -> OpenObserve for logs and traces, and
Tetragon/KubeArmor for runtime security events. Cover the agent/gateway split,
auto-instrumentation for traces, the servicegraph service map, why the collector
does not collect metrics, OpenObserve licensing against the Apache-2.0 OpenSearch
alternative, the deprecated filebeat and loki-stack charts, and the rule that
vendored subcharts are configured from the wrapper values.
@ashish1099
ashish1099 force-pushed the docs/observability branch from cebe743 to 5375fa4 Compare July 12, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant