RFC 0006: Native Histogram Support

[samsond]

Add RFC proposing Phase 1 native histogram support in the PromQL engine.

Phase 1 approves a narrow slice:

• Carry native histogram payloads through evaluator/function boundary
• Implement histogram_count, histogram_sum, histogram_avg projection functions
• Validate with unit tests and promqltest

Storage encoding (RecordType::Histogram), range semantics, HTTP serialization, and full quantile/fraction support are deferred to Phases 2-5.

Defines cross-RFC dependencies with RFC 0001 (storage), RFC 0003 (public types), and RFC 0005 (PromQL functions). Aligns with Prometheus evaluator FloatHistogram model for exponential-bucket native histograms.

Fixes #102

Checklist

• Tests added/updated
• cargo fmt and cargo clippy pass
• Documentation updated (if applicable)

[agavra]

A few comments on scope before I review the impl details.

[apurvam]

Hi @samsond

Thanks for the RFC. Reading through this, I'm wondering if it makes sense to try to prototype an end-to-end solution here to make sure we are addressing the real problems with histograms in the best possible way.

Specifically, the Prometheus design doc states the problems native histograms are solving as follows:

The problems result in the following practices:

• To prevent Histograms from taking most of the cardinality budget, users tend to
  have very few buckets per Histogram (3–10), leading to huge errors in quantile
  estimation (unless the quantile value falls into a carefully selected value range
  with a few narrow buckets).
• For the same reason, users shy away from partitioning Histograms. It is common
  to have a counter for HTTP requests that is partitioned along many label
  dimensions (status code, method, path, …) and then one latency Histogram for all
  HTTP requests combined, or only partitioned by a subset of those dimensions.
  This is not only tedious during instrumentation, it also restricts the questions that
  a user can later ask with PromQL (e.g. what’s the latency distribution of 2xx vs.
  5xx).
• Users have to know in advance, during instrumenting their code, what the
  “interesting” bucket boundaries will be (SLO thresholds, Apdex target latencies,
  etc.). That alone is problematic, but it also requires a change of the bucket layout
  if the boundaries of interest ever change. Furthermore, a bucket layout change is
  not only tedious to conduct, it also makes meaningful queries spanning Histogram
  metrics from before and after the transition impossible (with the exception of a
  compatible bucket layout change, which can only be done under specific
  circumstances with a lot of care).

A solution addressing the problems needs to meet the following requirements:

• No (or minimal) configuration during instrumentation, in particular no
  configuration that would require knowledge about the expected value ranges of
  observations and what bucket boundaries will be of particular interest later.
• Quantile and percentage estimations with a small error margin across the entire
  range of observations, including the long tail of latency distributions.
• Enable partitioning of Histograms similar to what is currently done with plain
  counters. Histograms will always be more expensive than plain counters, but the
  gap needs to be much smaller than it is now.

I think that's a very specific set of goals. Evaluating our existing system to find it's limitations on that spectrum would be a good place to start. Then prototyping native histograms will allow us to evaluate a design far more robustly.

My big worry about a series of large RFCs which only have a payoff in the distant future is that a) we may never get there, and b) if we do we will just call it a win just because we reached the end of the journey.

That dynamic plays out very often in open source projects. It was probably necessary to do it that way when code was so expensive to write. But testing the limits of current histograms and then prototyping solutions that address those limits is much cheaper now. doing so will allow us to compare ourselves against victoria metrics / prometheus / other implementations, and then choose a path more wisely.

I'd love your thoughts on such an approach.

[samsond]

Hi @samsond

Thanks for the RFC. Reading through this, I'm wondering if it makes sense to try to prototype an end-to-end solution here to make sure we are addressing the real problems with histograms in the best possible way.

Specifically, the Prometheus design doc states the problems native histograms are solving as follows:

The problems result in the following practices:

• To prevent Histograms from taking most of the cardinality budget, users tend to
  have very few buckets per Histogram (3–10), leading to huge errors in quantile
  estimation (unless the quantile value falls into a carefully selected value range
  with a few narrow buckets).
• For the same reason, users shy away from partitioning Histograms. It is common
  to have a counter for HTTP requests that is partitioned along many label
  dimensions (status code, method, path, …) and then one latency Histogram for all
  HTTP requests combined, or only partitioned by a subset of those dimensions.
  This is not only tedious during instrumentation, it also restricts the questions that
  a user can later ask with PromQL (e.g. what’s the latency distribution of 2xx vs.
  5xx).
• Users have to know in advance, during instrumenting their code, what the
  “interesting” bucket boundaries will be (SLO thresholds, Apdex target latencies,
  etc.). That alone is problematic, but it also requires a change of the bucket layout
  if the boundaries of interest ever change. Furthermore, a bucket layout change is
  not only tedious to conduct, it also makes meaningful queries spanning Histogram
  metrics from before and after the transition impossible (with the exception of a
  compatible bucket layout change, which can only be done under specific
  circumstances with a lot of care).

A solution addressing the problems needs to meet the following requirements:

• No (or minimal) configuration during instrumentation, in particular no
  configuration that would require knowledge about the expected value ranges of
  observations and what bucket boundaries will be of particular interest later.
• Quantile and percentage estimations with a small error margin across the entire
  range of observations, including the long tail of latency distributions.
• Enable partitioning of Histograms similar to what is currently done with plain
  counters. Histograms will always be more expensive than plain counters, but the
  gap needs to be much smaller than it is now.

I think that's a very specific set of goals. Evaluating our existing system to find it's limitations on that spectrum would be a good place to start. Then prototyping native histograms will allow us to evaluate a design far more robustly.

My big worry about a series of large RFCs which only have a payoff in the distant future is that a) we may never get there, and b) if we do we will just call it a win just because we reached the end of the journey.

That dynamic plays out very often in open source projects. It was probably necessary to do it that way when code was so expensive to write. But testing the limits of current histograms and then prototyping solutions that address those limits is much cheaper now. doing so will allow us to compare ourselves against victoria metrics / prometheus / other implementations, and then choose a path more wisely.

I'd love your thoughts on such an approach.

Thanks, this is a helpful framing.

I agree with the underlying concern: if we only discuss native histograms abstractly, it becomes too easy to mistake “we completed the planned phases” for “we solved the right problem.” I also agree that Prometheus’s motivation is quite specific: native histograms are not just “another sample type,” they are meant to address concrete limitations around bucket resolution, partitioning cost, and the need to lock in bucket boundaries too early.

I think your suggested approach is a good one, or at least the right correction to what I wrote so far. Before treating native histograms as the default destination, we should be more explicit about:

• what limitations we see in OpenData’s current float/classic histogram model,
• which of Prometheus’s target problems actually matter for our users and expected workloads,
• how OTEL histogram inputs map onto those needs in practice, and
• what an end-to-end prototype tells us that an evaluator-only slice cannot.

My intent with the phased RFC was to avoid prematurely committing us to full storage/ingest/API work, but I think your point is that there is a risk in the other direction too: we can end up validating architecture in pieces without validating that the overall direction is worth it.

So I think there are really two distinct questions here:

• do we want a narrow architectural slice to understand what supporting histogram-valued samples would entail in the evaluator/runtime, and
• do we want an end-to-end prototype or design spike that tests whether native histograms actually solve an important problem for OpenData better than the alternatives?

I do not think those are in conflict. In fact, the RFC probably needs to make clearer that the architecture slice is not sufficient evidence on its own. We likely need some combination of:

• a gap analysis of current/classic histogram limitations in OpenData,
• a comparison of Prometheus native histograms, OTEL histograms, and other implementations such as VictoriaMetrics,
• and a small prototype that lets us evaluate ingest, storage shape, query ergonomics, and cost profile together.

That would also make the success criteria much healthier.

[apurvam]

@samsond You're right in that I don't think doing a prototype / gap analysis / comparison etc is in conflict with what's proposed here. I do think we need to do the former before committing to an incremental plan that builds native histograms though.

So I would say that we should probably take a step back, do some meaningful benchmarks on the current state. Compare that with what prometheus can do with native histograms, and then agree on whether we want to close the gap. If we do, then we should prototype a solution that gets us to where we want to be. Once we have that prototype, we can create incremental RFCs / PRs like this one.