Add OpenObserve-compatible Prometheus query mode for KRR resource recommendations

[kayahk]

I’m trying to run KRR against OpenObserve’s Prometheus-compatible API to generate Kubernetes resource recommendations from metrics already ingested into OpenObserve.

Example endpoint shape:

krr simple \
  --context dev1 \
  --namespace krr-demo \
  --prometheus-url https://openobserve.example.com/api/dev1/prometheus \
  --prometheus-auth-header "Basic <redacted>"

Authentication works: KRR connects successfully to the OpenObserve Prometheus API.

The scan then fails because KRR’s default Prometheus metric loaders use PromQL subqueries such as:

quantile_over_time(
  0.95,
  max(rate(container_cpu_usage_seconds_total{...}[75s])) by (container, pod, job)
  [14d:75s]
)

max_over_time(
  max(container_memory_working_set_bytes{...}) by (container, pod, job)
  [14d:75s]
)

count_over_time(
  max(container_cpu_usage_seconds_total{...}) by (container, pod, job)
  [14d:75s]
)

OpenObserve rejects these with HTTP 400:

This feature is not implemented: Unsupported subquery, the return value should have been a matrix but got "scalar"

This means KRR can connect to OpenObserve, but cannot currently calculate recommendations with the default strategy.

Proposed approach:

Add an explicit provider option, for example:

--prometheus-provider openobserve

For this provider, KRR could avoid PromQL subqueries by querying simpler range expressions and doing the final aggregation locally:

• CPU percentile: query max(rate(container_cpu_usage_seconds_total{...}[step])) by (...) via query_range, then calculate the percentile in Python.
• Memory max: query max(container_memory_working_set_bytes{...}) by (...) via query_range, then calculate the max in Python.
• Sample counts: count returned range samples locally instead of using count_over_time(...).
• This would preserve existing default Prometheus behavior while allowing OpenObserve users to run the same recommendation logic.

I have a draft implementation in a fork that adds this as --prometheus-provider openobserve and keeps the default provider unchanged.