[bug] Instrumentation.StackExchangeRedis High memory

[jojo-Fish]

Component

OpenTelemetry.Instrumentation.StackExchangeRedis

Package Version

Package Name	Version
OpenTelemetry.Exporter.Console	1.15.2
OpenTelemetry.Extensions.Hosting	1.15.2
OpenTelemetry.Instrumentation.AspNetCore	1.15.1
OpenTelemetry.Instrumentation.Http	1.15.0
OpenTelemetry.Instrumentation.StackExchangeRedis	1.15.0-beta.1

Runtime Version

net10.0

Description

code demo

when this test action executed,the memory become High memory usage,even got more than 2.5GB usage

Steps to Reproduce

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.DataProtection.KeyManagement;
using OpenTelemetry;
using OpenTelemetry.Trace;
using StackExchange.Redis;

var builder = WebApplication.CreateBuilder(args);

var connection = ConnectionMultiplexer.Connect("localhost:6693,abortConnect=false,keepAlive=30");
builder.Services.AddSingleton<IConnectionMultiplexer>(connection);
builder.Services.AddOpenTelemetry()
    .WithTracing(tracingBuilder =>
    {
        tracingBuilder.SetSampler(new AlwaysOnSampler());
        tracingBuilder
            .AddHttpClientInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRedisInstrumentation(connection, opt => opt.FlushInterval = TimeSpan.FromSeconds(2))
            .AddConsoleExporter();
    });
var app = builder.Build();
app.MapGet("/test", async (IConnectionMultiplexer connect) =>
{
    var db = connection.GetDatabase();
    var counter = 0;
    await Parallel.ForEachAsync(
        Enumerable.Range(0, 200_0000),
        new ParallelOptions { MaxDegreeOfParallelism = 10 },
        async (i, _) =>
        {
            var key = $"text4_{i}";

            await db.HashSetAsync(key, i, 1);
            await db.KeyExpireAsync(key, TimeSpan.FromHours(3));

            Interlocked.Increment(ref counter);
        });
    return $"Total: {counter}";
});
await app.RunAsync();

Expected Result

when i disable AddRedisInstrumentation,the memory usage is very low,the Instrumentation.Stackexchange.redis has some memroy problem?

Actual Result

very hign memory usage

Additional Context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[martincostello]

Thanks for opening this issue. To make this issue more actionable you:

• Simplify the example and make it self-contained? As written it:
  • Connects to servers we won't have access to (and it includes your password)
  • Uses the same sampler three times
  • Uses both AlwaysOn and AlwaysOff samplers
  • Is missing various parts (like setting up ASP.NET Core) that we can't just paste and run to debug
• Provide some concrete numbers of what exactly you're observing and over what kind of period of time and number of requests

[jojo-Fish]

@martincostello Thanks for reply.the example has edited.

here is the profiling of memory-usage
start

running for some time

[martincostello]

I'd adapted your repro to something a little bit more realistic - an HTTP request that does 4,000,000 Redis operations in a single HTTP request isn't realistic, and will also cause a large number of profiles to be buffered in memory as the parent activity (the HTTP request) is ongoing for as long as that takes before the traces are flushed to the exporter.

With tracing disabled, memory seems to reach a steady state of ~385MB after ~2m:20s.

With tracing enabled, memory reaches a steady state of ~442MB after ~50s.

This isn't showing an obvious issue to me - it just shows that tracing has an overhead and that it gets to steady-state faster, which is expected because more work needs to be done with tracing enabled.

For the code below, the overhead is ~57MB, or an additional ~14%.

If there is an issue, we need something a bit more realistic and targeted that shows there's a clear excessive overhead and/or a memory leak.

Repro Code

using OpenTelemetry.Trace;
using StackExchange.Redis;

bool enableTracing = false;

var builder = WebApplication.CreateBuilder(args);

using var connection = ConnectionMultiplexer.Connect("localhost,abortConnect=false,keepAlive=30");
builder.Services.AddSingleton<IConnectionMultiplexer>(connection);

if (enableTracing)
{
    builder.Services.AddOpenTelemetry()
        .WithTracing(tracingBuilder =>
        {
            tracingBuilder.SetSampler(new AlwaysOnSampler());
            tracingBuilder
                .AddRedisInstrumentation(connection, opt => opt.FlushInterval = TimeSpan.FromSeconds(2))
                .AddConsoleExporter();
        });
}

var app = builder.Build();

app.MapGet("/", async (IConnectionMultiplexer connect, CancellationToken cancellationToken) =>
{
    var db = connection.GetDatabase();
    var timeout = TimeSpan.FromHours(3);

    for (var i = 0; i < 20 && !cancellationToken.IsCancellationRequested; i++)
    {
        var key = $"key-{i}";
        await db.HashSetAsync(key, i, 1);
        await db.KeyExpireAsync(key, timeout);
    }

    return TypedResults.Ok();
});

await app.StartAsync();

var server = app.Services.GetRequiredService<Microsoft.AspNetCore.Hosting.Server.IServer>();
var addresses = server.Features.Get<Microsoft.AspNetCore.Hosting.Server.Features.IServerAddressesFeature>();

var baseAddress = addresses!.Addresses
    .Select((p) => new Uri(p))
    .Last();

using var client = new HttpClient();
using var cts = new CancellationTokenSource();

Console.CancelKeyPress += (_, e) =>
{
    cts.Cancel();
    e.Cancel = true;
};

while (!cts.IsCancellationRequested)
{
    try
    {
        await client.GetStringAsync(baseAddress, cts.Token);
    }
    catch (OperationCanceledException) when (cts.IsCancellationRequested)
    {
        break;
    }
}

await app.StopAsync();

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>$(DefaultTargetFrameworkForExampleApps)</TargetFramework>
    <OutputType>Exe</OutputType>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="OpenTelemetry.Exporter.Console" />
    <PackageReference Include="OpenTelemetry.Exporter.InMemory" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="$(RepoRoot)\src\OpenTelemetry.Instrumentation.StackExchangeRedis\OpenTelemetry.Instrumentation.StackExchangeRedis.csproj" />
  </ItemGroup>
</Project>

Disabled

Enabled

[jojo-Fish]

@martincostello Thanks for the reply. In fact,an HTTP request that does 4,000,000 Redis operations in a single HTTP request isn't realistic。BUT In some cases, our business requirements mean we need to run cache warm-up tasks using jobs. As a result, a single job may execute a loop that issues millions of Redis commands, like in my demo. Once Redis instrumentation is enabled, we see memory usage gradually increase. Is this expected and normal behavior?