Default OpenTelemetry histogram metric boundaries don't fit metrics with a unit of seconds

[brather1ng]

Describe the suggested improvement

Is your improvement related to a problem? Please describe.

Metrics like nservicebus.messaging.handler_time are using the default OpenTelemetry histogram buckets, at least those in https://github.com/Particular/NServiceBus/blob/master/src/NServiceBus.Core/Pipeline/Incoming/IncomingPipelineMetrics.cs.

The default buckets are [ 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000 ], see the Microsoft docs. For measurements in seconds, these don't fit. This seems to be an accepted discrepancy in the OpenTelemetry SDK, see e.g. open-telemetry/opentelemetry-dotnet#4797.

Describe the suggested solution

Provide custom HistogramBucketBoundaries when defining histogram metrics that better match the unit. opentelemetry-dotnet uses [ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ] (default divided by 1000), but the upper end might be too low NServiceBus use cases.

Describe alternatives you've considered

Additional Context

Example: If message handling usually takes below 5 seconds, everything goes into the same bucket and the metric only provides information on extreme outliers.

# TYPE nservicebus_messaging_handler_time_seconds histogram
# HELP nservicebus_messaging_handler_time_seconds The time in seconds for the execution of the business code.
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="0"} 0
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="5"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="10"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="25"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="50"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="75"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="100"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="250"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="500"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="750"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="1000"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="2500"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="5000"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="7500"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="10000"} 5
nservicebus_messaging_handler_time_seconds_bucket{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint",le="+Inf"} 5
nservicebus_messaging_handler_time_seconds_sum{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint"} 2.5933935999999997
nservicebus_messaging_handler_time_seconds_count{otel_scope_name="NServiceBus.Core.Pipeline.Incoming",otel_scope_version="0.2.0",error_type="System.Net.Http.HttpRequestException",execution_result="failure",nservicebus_discriminator="",nservicebus_message_handler_type="RedactedHandler",nservicebus_message_type="RedactedMessageType",nservicebus_queue="RedactedMessageEndpoint"} 5

[stevedowling]

Thank you for your contribution. This has been added to our backlog.