Skip to content

Disable OpenObserve infra-metrics + move collector overrides to the wrapper layer - #75

Draft
ashish1099 wants to merge 1 commit into
masterfrom
feat/openobserve-no-metrics
Draft

Disable OpenObserve infra-metrics + move collector overrides to the wrapper layer#75
ashish1099 wants to merge 1 commit into
masterfrom
feat/openobserve-no-metrics

Conversation

@ashish1099

@ashish1099 ashish1099 commented Jul 12, 2026

Copy link
Copy Markdown
Member

Why

Two things, both rooted in one rule: <app>/charts/ is the vendored upstream subchart and must not carry local customization — a re-vendor silently drops it. Customization belongs in the first-layer wrapper (argocd-helm-charts/openobserve/values.yaml).

  1. OpenObserve must not ingest infra metrics. Metrics + alerting live only in kube-prometheus.
  2. PR Default openobserve-collector exporter to in-cluster router #72's endpoint fix slipped into the upstream subchart — it would have reverted to OpenObserve Cloud (api.openobserve.ai) on the next re-vendor, silently shipping cluster telemetry off-cluster.

Changes

1. Wrapper overrides (first layer) — openobserve/values.yaml

openobserve-collector:
  exporters:                                  # pinned here so it survives re-vendoring
    otlphttp/openobserve:
      endpoint: http://openobserve-router.openobserve.svc.cluster.local:5080/api/default
    otlphttp/openobserve_k8s_events:
      endpoint: http://openobserve-router.openobserve.svc.cluster.local:5080/api/default
  agent:                                      # logs-only
    receivers: {hostmetrics: null, kubeletstats: null}
    service: {pipelines: {metrics: null}}
  gateway:                                    # traces + k8s events + service map
    receivers: {prometheus: null}
    service:
      pipelines:
        metrics:
          receivers: [servicegraph]
  kube-state-metrics:
    enabled: false
  • Agent → logs-only. hostmetrics + kubeletstats dropped (duplicated node-exporter / kubelet-cAdvisor).
  • Gateway → traces + k8s events, plus the service map. The prometheus scrape and the otlp metrics receiver are dropped (real infra metrics). The servicegraph connector is KEPT — it's derived from traces (service map / RED metrics), exists nowhere in Prometheus, and is part of the tracing experience. The metrics pipeline is narrowed to receivers: [servicegraph]; the traces pipeline is untouched and still feeds the connector.
  • Bundled kube-state-metrics disabled — kube-prometheus already ships KSM.

Uses Helm null-coalescing to delete upstream keys — the convention already used here (e.g. cert-manager.io/issuer: null).

2. Subchart restored to pristine upstream (second layer)

Reverts PR #72's edits to charts/openobserve-collector/: exporter endpoints back to https://api.openobserve.ai/api/default/, chart version back to 0.4.4. Net runtime behaviour is unchanged — the collector still exports to the in-cluster router; the setting just lives in the layer that persists.

Metric coverage is unaffected

Signal Prometheus source
node/host CPU·mem·disk·net node-exporter
per-pod/container usage kubelet /metrics/cadvisor
k8s object state kube-state-metrics

Wrapper chart bumped 0.13.5 → 0.13.6.

@ashish1099
ashish1099 force-pushed the feat/openobserve-no-metrics branch from 1c5a38c to f54e4a9 Compare July 12, 2026 02:44
@ashish1099 ashish1099 changed the title Stop shipping host metrics from openobserve-collector to OpenObserve Remove all metrics ingestion into OpenObserve (metrics stay in kube-prometheus) Jul 12, 2026
@ashish1099 ashish1099 closed this Jul 12, 2026
@ashish1099
ashish1099 force-pushed the feat/openobserve-no-metrics branch from f54e4a9 to d47b766 Compare July 12, 2026 02:50
@ashish1099 ashish1099 changed the title Remove all metrics ingestion into OpenObserve (metrics stay in kube-prometheus) Disable all OpenObserve metrics ingestion via wrapper overrides Jul 12, 2026
@ashish1099 ashish1099 reopened this Jul 12, 2026
@ashish1099 ashish1099 changed the title Disable all OpenObserve metrics ingestion via wrapper overrides Disable OpenObserve metrics + move collector overrides to the wrapper layer Jul 12, 2026
@ashish1099 ashish1099 changed the title Disable OpenObserve metrics + move collector overrides to the wrapper layer Disable OpenObserve infra-metrics + move collector overrides to the wrapper layer Jul 12, 2026
…rapper

OpenObserve stores logs, traces and k8s/security events. Infra metrics and
alerting stay in kube-prometheus, which already ships node-exporter,
kubelet/cAdvisor scraping and kube-state-metrics.

Configure the collector from the first-layer wrapper values.yaml instead of the
vendored openobserve-collector subchart, using Helm null-coalescing:

- agent: drop the hostmetrics and kubeletstats receivers and the metrics
  pipeline, leaving a logs-only collector
- gateway: drop the prometheus scrape and the otlp metrics receiver; narrow the
  metrics pipeline to the servicegraph connector, which is derived from traces
  and has no Prometheus equivalent
- disable the bundled kube-state-metrics
- pin the OTLP exporter to the in-cluster OpenObserve router

Restore charts/openobserve-collector to upstream (exporter endpoints and chart
version 0.4.4); the subchart tracks upstream and must not carry local changes.
Runtime behaviour is unchanged.

Bump the openobserve wrapper chart 0.13.5 -> 0.13.6.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant