Does OpenTelemetry support a custom Http handler (DelegatingHandler) for logs/metrics/traces?

[pcfountain77]

We're implementing OpenTelemetry in our Azure-hosted .NET apps. We currently use app insights for log analytics, but the plan is to send OTel logs to a collector that will then be ingested by SumoLogic. By examining our logs in Azure app insights, we've discovered that around 10% of our metrics calls to the collector are failing. In an attempt to gather more information, we used AI tools to implement a DelegatingHandler that logs any failed calls. However, we didn't see any additional logs for failed calls. So I'm curious if the OpenTelemetry libraries even support this approach.

Here is our setup for the DelegatingHandler:

        // Use dedicated named clients so OTLP metric/trace exports can log transport failures
        // and avoid reusing minute-idle connections across export cycles.
        services.AddHttpClient("OtlpMetricExporter")
            .ConfigurePrimaryHttpMessageHandler(CreateOtlpExporterHandler)
            .AddHttpMessageHandler<OtlpExportLoggingHandler>();

        services.AddHttpClient("OtlpTraceExporter")
            .ConfigurePrimaryHttpMessageHandler(CreateOtlpExporterHandler)
            .AddHttpMessageHandler<OtlpExportLoggingHandler>();

    private static SocketsHttpHandler CreateOtlpExporterHandler() => new()
    {
        AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip,
        ConnectTimeout = TimeSpan.FromSeconds(10),
        PooledConnectionIdleTimeout = TimeSpan.FromSeconds(20),
        PooledConnectionLifetime = TimeSpan.FromMinutes(5),
    };

And here is the handler itself:

using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;

namespace MyApp.API.OpenTelemetry;

public sealed class OtlpExportLoggingHandler(ILogger<OtlpExportLoggingHandler> logger) : DelegatingHandler
{
    async protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var stopwatch = Stopwatch.StartNew();

        try
        {
            var response = await base.SendAsync(request, cancellationToken);

            if (!response.IsSuccessStatusCode)
            {
                logger.LogWarning(
                    "OTLP export returned {StatusCode} from {RequestUri} after {ElapsedMilliseconds} ms.",
                    (int)response.StatusCode,
                    request.RequestUri,
                    stopwatch.ElapsedMilliseconds);
            }

            return response;
        }
        catch (Exception ex)
        {
            logger.LogError(
                ex,
                "OTLP export to {RequestUri} failed after {ElapsedMilliseconds} ms. InnerExceptionType={InnerExceptionType} InnerExceptionMessage={InnerExceptionMessage}",
                request.RequestUri,
                stopwatch.ElapsedMilliseconds,
                ex.InnerException?.GetType().FullName,
                ex.InnerException?.Message);

            throw;
        }
    }
}

Again, this code was AI-generated. From my understanding, this will only work if the OpenTelemetry libraries are using named HttpClients with the specific names OtlpMetricExporter and OtlpTraceExporter. I'm not sure if this is the case. Any suggestions or feedback is welcome. Thank you!

[martincostello]

You can either assign a delegate to a method to create HttpClient instances yourself:

opentelemetry-dotnet/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpExporterOptions.cs

Lines 155 to 165 in 8bee1f4

	/// <inheritdoc/>
	public Func<HttpClient> HttpClientFactory
	{
	get => this.httpClientFactory ?? this.DefaultHttpClientFactory;
	set
	{
	Guard.ThrowIfNull(value);
	this.httpClientFactory = value;
	}
	}

Or there's an HttpClientFactory integration (code).