Skip to content

[Instrumentation.Kusto] Add instrumentor for Azure Kusto#3591

Open
MattKotsenas wants to merge 26 commits intoopen-telemetry:mainfrom
MattKotsenas:feature/kusto-instrumentor
Open

[Instrumentation.Kusto] Add instrumentor for Azure Kusto#3591
MattKotsenas wants to merge 26 commits intoopen-telemetry:mainfrom
MattKotsenas:feature/kusto-instrumentor

Conversation

@MattKotsenas
Copy link
Copy Markdown
Contributor

@MattKotsenas MattKotsenas commented Dec 8, 2025

Fixes #3575

Changes

Adds an initial implementation of traces and metrics for any client built on top of the Microsoft.Azure.Kusto.Cloud.Platform package (namely Microsoft.Azure.Kusto.Data and Microsoft.Azure.Kusto.Ingest).

The library generally follows the semantic conventions for dbs for both traces and metrics, importantly, it does both query summarization and sanitization. It does not support adding query parameters as tags, as those aren't available in Kusto's internal custom tracing framework. Test coverage includes unit tests, benchmarks both for end-to-end instrumentation and query processing specific, and integration tests using the existing pattern of Testcontainers.

The package is not strong-name signed because it uses Microsoft.Azure.Kusto.Language for query processing, which also is not strong-name signed.

Merge requirement checklist

  • CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
  • Unit tests added/updated
  • Appropriate CHANGELOG.md files updated for non-trivial changes
  • Changes in public API reviewed (if applicable)

@MattKotsenas MattKotsenas requested a review from a team as a code owner December 8, 2025 19:37
@github-actions github-actions bot added infra Infra work - CI/CD, code coverage, linters dependencies Pull requests that update a dependency file perf Performance related labels Dec 8, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 8, 2025

Codecov Report

❌ Patch coverage is 90.99099% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.06%. Comparing base (d1717a8) to head (3a97216).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...n.Kusto/Implementation/KustoTraceRecordListener.cs 83.92% 18 Missing ⚠️
.../Implementation/KustoInstrumentationEventSource.cs 0.00% 11 Missing ⚠️
...on.Kusto/Implementation/TruncatingStringBuilder.cs 97.22% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3591      +/-   ##
==========================================
+ Coverage   72.75%   73.06%   +0.31%     
==========================================
  Files         458      473      +15     
  Lines       17876    18209     +333     
==========================================
+ Hits        13006    13305     +299     
- Misses       4870     4904      +34     
Flag Coverage Δ
unittests-Contrib.Shared.Tests 89.76% <ø> (ø)
unittests-Exporter.Geneva 54.82% <ø> (ø)
unittests-Exporter.InfluxDB 95.81% <ø> (ø)
unittests-Exporter.Instana 74.86% <ø> (-0.19%) ⬇️
unittests-Exporter.OneCollector 94.61% <ø> (ø)
unittests-Extensions 90.65% <ø> (ø)
unittests-Extensions.Enrichment 100.00% <ø> (ø)
unittests-Extensions.Enrichment.AspNetCore 86.27% <ø> (ø)
unittests-Extensions.Enrichment.Http 94.33% <ø> (ø)
unittests-Instrumentation.AWS 83.54% <ø> (ø)
unittests-Instrumentation.AspNet 77.37% <ø> (ø)
unittests-Instrumentation.AspNetCore 70.44% <ø> (ø)
unittests-Instrumentation.Cassandra 23.52% <ø> (ø)
unittests-Instrumentation.ConfluentKafka 39.83% <ø> (ø)
unittests-Instrumentation.ElasticsearchClient 80.60% <ø> (ø)
unittests-Instrumentation.EntityFrameworkCore 80.80% <ø> (ø)
unittests-Instrumentation.EventCounters 77.27% <ø> (ø)
unittests-Instrumentation.GrpcCore 91.42% <ø> (ø)
unittests-Instrumentation.GrpcNetClient 73.78% <ø> (ø)
unittests-Instrumentation.Hangfire 86.05% <ø> (ø)
unittests-Instrumentation.Http 74.62% <ø> (ø)
unittests-Instrumentation.Kusto 90.99% <90.99%> (?)
unittests-Instrumentation.Owin 88.62% <ø> (ø)
unittests-Instrumentation.Process 100.00% <ø> (ø)
unittests-Instrumentation.Quartz 78.76% <ø> (ø)
unittests-Instrumentation.Runtime 100.00% <ø> (ø)
unittests-Instrumentation.ServiceFabricRemoting 34.68% <ø> (ø)
unittests-Instrumentation.SqlClient 85.74% <ø> (ø)
unittests-Instrumentation.StackExchangeRedis 71.98% <ø> (ø)
unittests-Instrumentation.Wcf 79.68% <ø> (ø)
unittests-OpAmp.Client 79.11% <ø> (-0.38%) ⬇️
unittests-PersistentStorage 69.83% <ø> (ø)
unittests-Resources.AWS 74.34% <ø> (ø)
unittests-Resources.Azure 85.31% <ø> (ø)
unittests-Resources.Container 67.34% <ø> (ø)
unittests-Resources.Gcp 71.42% <ø> (ø)
unittests-Resources.Host 72.26% <ø> (ø)
unittests-Resources.OperatingSystem 76.98% <ø> (ø)
unittests-Resources.Process 100.00% <ø> (ø)
unittests-Resources.ProcessRuntime 79.59% <ø> (ø)
unittests-Sampler.AWS 94.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...entation/InstrumentationHandleManagerExtensions.cs 100.00% <100.00%> (ø)
....Kusto/Implementation/KustoActivitySourceHelper.cs 100.00% <100.00%> (ø)
...ation.Kusto/Implementation/KustoInstrumentation.cs 100.00% <100.00%> (ø)
...rumentation.Kusto/Implementation/KustoProcessor.cs 100.00% <100.00%> (ø)
...ntation.Kusto/Implementation/KustoStatementInfo.cs 100.00% <100.00%> (ø)
...rumentation.Kusto/Implementation/SpanExtensions.cs 100.00% <100.00%> (ø)
...tion.Kusto/Implementation/TraceRecordExtensions.cs 100.00% <100.00%> (ø)
...entation.Kusto/Implementation/TraceRecordParser.cs 100.00% <100.00%> (ø)
...entation.Kusto/KustoMeterInstrumentationOptions.cs 100.00% <100.00%> (ø)
...entation.Kusto/KustoTraceInstrumentationOptions.cs 100.00% <100.00%> (ø)
... and 5 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@MattKotsenas MattKotsenas force-pushed the feature/kusto-instrumentor branch 3 times, most recently from 0aef4ac to 77819be Compare December 8, 2025 21:28
@github-actions
Copy link
Copy Markdown
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added Stale and removed Stale labels Dec 18, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 3, 2026

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jan 3, 2026
@MattKotsenas
Copy link
Copy Markdown
Contributor Author

Please do not close due to staleness. There won't be much movement on this over the holidays, but I'll return to this in the new year. Thanks!

@github-actions github-actions bot removed the Stale label Jan 6, 2026
@github-actions
Copy link
Copy Markdown
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jan 14, 2026
@aicodeguard
Copy link
Copy Markdown

Code Review – Key Findings

Summary:
Two critical issues were identified in the Kusto instrumentation implementation. The most significant is a performance risk due to synchronous semantic analysis of every query on the hot path. Additionally, a logic bug in the query sanitizer alters the meaning of queries involving unary operators (e.g., negation), leading to misleading telemetry.

Issues

  1. [Performance] Synchronous Semantic Analysis on Hot Path

    • File: src/OpenTelemetry.Instrumentation.Kusto/Implementation/KustoProcessor.cs
    • Lines: 60
    • Details: The Process method calls KustoCode.ParseAndAnalyze synchronously for every query when summarization is enabled (which is the default). This invokes the full Kusto language service semantic analyzer, which is computationally heavy and will add significant latency to every database request.
    • Fix: Avoid full semantic analysis on the request processing path. Use KustoCode.Parse (syntax only) if possible, or disable query summarization by default to avoid performance regression.
  2. [Bug] Incorrect Sanitization of Unary Operators

    • File: src/OpenTelemetry.Instrumentation.Kusto/Implementation/KustoProcessor.cs
    • Lines: 129-133
    • Details: The SanitizerVisitor unconditionally removes the operator from all prefix unary expressions. This changes the logic of the query in the trace; for example, where !IsReady is recorded as where IsReady.
      public override void VisitPrefixUnaryExpression(PrefixUnaryExpression node)
      {
          this.edits.Add(CreateRemoval(node.Operator)); // Removes '!', '-', '~' etc.
          base.VisitPrefixUnaryExpression(node);
      }
    • Fix: Modify VisitPrefixUnaryExpression to check the operator type and the operand. Only remove the operator if it is part of a literal value being sanitized (e.g., a negative number), but preserve logical operators like ! or ~.

@rajkumar-rangaraj rajkumar-rangaraj added keep-open Prevents issues and pull requests being closed as stale and removed Stale labels Jan 15, 2026
@MattKotsenas MattKotsenas force-pushed the feature/kusto-instrumentor branch from a651651 to 9a5e236 Compare March 10, 2026 18:53
@MattKotsenas
Copy link
Copy Markdown
Contributor Author

/cc @rajkumar-rangaraj, ready to restart the review process

@MattKotsenas
Copy link
Copy Markdown
Contributor Author

  1. [Performance] Synchronous Semantic Analysis on Hot Path

    • File: src/OpenTelemetry.Instrumentation.Kusto/Implementation/KustoProcessor.cs
    • Lines: 60
    • Details: The Process method calls KustoCode.ParseAndAnalyze synchronously for every query when summarization is enabled (which is the default). This invokes the full Kusto language service semantic analyzer, which is computationally heavy and will add significant latency to every database request.
    • Fix: Avoid full semantic analysis on the request processing path. Use KustoCode.Parse (syntax only) if possible, or disable query summarization by default to avoid performance regression.

The semantic analysis is required when summarizing in order to flag table-like elements. Per the benchmarks, I'm seeing single digit microseconds for summarization + sanitization. The slightly bigger issue we might run into is the allocations that the KustoCode parser allocates. We can avoid both with a query cache, which I didn't add pre-emptively because a cache without an expiration policy is a memory leak, but am happy to revisit.

  1. [Bug] Incorrect Sanitization of Unary Operators

    • File: src/OpenTelemetry.Instrumentation.Kusto/Implementation/KustoProcessor.cs
    • Lines: 129-133
    • Details: The SanitizerVisitor unconditionally removes the operator from all prefix unary expressions. This changes the logic of the query in the trace; for example, where !IsReady is recorded as where IsReady.
      public override void VisitPrefixUnaryExpression(PrefixUnaryExpression node)
      {
          this.edits.Add(CreateRemoval(node.Operator)); // Removes '!', '-', '~' etc.
          base.VisitPrefixUnaryExpression(node);
      }
    • Fix: Modify VisitPrefixUnaryExpression to check the operator type and the operand. Only remove the operator if it is part of a literal value being sanitized (e.g., a negative number), but preserve logical operators like ! or ~.

I'm happy to do this if we feel it's really necessary, however from a sanitization perspective I'm not sure the complexity is worth the cost.

@rajkumar-rangaraj
Copy link
Copy Markdown
Member

The PR does not update .github.qkg1.topponent_owners.yml.

Maintaining code owners is an important requirement for this repo, and each component should have at least two owners.
Please add owners for the following:

 src/OpenTelemetry.Instrumentation.Kusto/:
   - owners
 test/OpenTelemetry.Instrumentation.Kusto.Tests/:
   - owners

@rajkumar-rangaraj
Copy link
Copy Markdown
Member

The PR puts options in OpenTelemetry.Trace and OpenTelemetry.Metrics:

  • OpenTelemetry.Trace.KustoTraceInstrumentationOptions
  • OpenTelemetry.Metrics.KustoMeterInstrumentationOptions

The predominant repo pattern (SqlClient, Http, etc.) puts options in the instrumentation's own
namespace:

  • OpenTelemetry.Instrumentation.SqlClient.SqlClientTraceInstrumentationOptions
  • OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions

Consider moving to OpenTelemetry.Instrumentation.Kusto.

@rajkumar-rangaraj
Copy link
Copy Markdown
Member

Verify.Xunit is not used anywhere else in the repo. This PR introduces it as a new test framework dependency plus
adds .gitignore entries (*.received.*) at the repo root level.

@Kielek / @martincostello What are your thoughts on it?

@MattKotsenas
Copy link
Copy Markdown
Contributor Author

The PR does not update .github.qkg1.topponent_owners.yml.

Maintaining code owners is an important requirement for this repo, and each component should have at least two owners. Please add owners for the following:

 src/OpenTelemetry.Instrumentation.Kusto/:
   - owners
 test/OpenTelemetry.Instrumentation.Kusto.Tests/:
   - owners

Who should I put here? I'm happy to put myself as one, but hadn't originally because I believe I need to be sponsored to do so (would be honored :)).

@martincostello
Copy link
Copy Markdown
Member

Verify.Xunit is not used anywhere else in the repo.

I'm happy to introduce it. I use it a lot myself, and we also use it in open-telemetry/opentelemetry-dotnet (added in open-telemetry/opentelemetry-dotnet#6567).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some other settings we should add for Verify, such as changes to .editorconfig settings. See here for some of the missing parts.

<PackageVersion Include="Microsoft.Data.SqlClient" Version="7.0.0" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Sqlite" Version="8.0.25" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.25" />

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

## Unreleased

For more details, please refer to the [README](README.md).
* Initial implementation.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Initial implementation.
* Initial implementation.
([#3591](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/pull/3591))

/// <summary>
/// Helper class to hold common properties used by Kusto instrumentation.
/// </summary>
internal static class KustoActivitySourceHelper
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this contains activities and meters, I would give it a different name.

{
private static readonly Lazy<ITraceListener> Listener = new(() =>
{
Environment.SetEnvironmentVariable("KUSTO_DATA_TRACE_REQUEST_BODY", "1");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we should be changing process env vars internally. If this is some internal implementation detail like how the Azure SDKs need (needed?) an opt-in to emit OTel metrics, I would instead document that the user is required to set that themselves.

It's also messy for tests as it changes the test environment.

{
services.Configure<KustoTraceInstrumentationOptions>(options =>
{
options.Enrich = (activity, record) => { enrichCalled = true; };
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
options.Enrich = (activity, record) => { enrichCalled = true; };
options.Enrich = (_, _) => enrichCalled = true;

/// <summary>
/// Class to hold the singleton instances used for Kusto instrumentation.
/// </summary>
internal static class KustoInstrumentation
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to design this without needing statics? Then the tests can be simpler too.

Comment on lines +23 to +24
=> new KustoBuilder(KustoImage)
.Build();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
=> new KustoBuilder(KustoImage)
.Build();
=> new KustoBuilder(KustoImage).Build();

Comment on lines +14 to +24
public DependencyInjectionConfigTests()
{
KustoInstrumentation.TraceOptions = new KustoTraceInstrumentationOptions();
KustoInstrumentation.MeterOptions = new KustoMeterInstrumentationOptions();
}

public void Dispose()
{
KustoInstrumentation.TraceOptions = new KustoTraceInstrumentationOptions();
KustoInstrumentation.MeterOptions = new KustoMeterInstrumentationOptions();
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The need to keep mutating statics feels like a smell to me.

Comment on lines +35 to +66
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddInMemoryExporter(activities)
.AddKustoInstrumentation(options =>
{
options.RecordQueryText = processQuery;
options.RecordQuerySummary = processQuery;
})
.Build();

using var meterProvider = Sdk.CreateMeterProviderBuilder()
.AddInMemoryExporter(metrics)
.AddKustoInstrumentation(options =>
{
options.RecordQueryText = processQuery;
options.RecordQuerySummary = processQuery;
})
.Build();

var kcsb = this.fixture.ConnectionStringBuilder;
using var queryProvider = KustoClientFactory.CreateCslQueryProvider(kcsb);

var crp = new ClientRequestProperties()
{
// Ensure a stable client ID for snapshots
ClientRequestId = Convert.ToBase64String(Encoding.UTF8.GetBytes(query)),
};

using var reader = queryProvider.ExecuteQuery("NetDefaultDB", query, crp);
reader.Consume();

tracerProvider.ForceFlush();
meterProvider.ForceFlush();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would wrap all these in an explicit using - we've had some test flakiness in the past where activities/metrics get registered after the flush, and then the asserts fail as something's missing. The extra dispose before asserting helps avoid that.

public static readonly string PackageVersion = Assembly.GetPackageVersion();

public static readonly string ActivitySourceName = AssemblyName.Name!;
public static readonly ActivitySource ActivitySource = new(ActivitySourceName, PackageVersion);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also include a schema URL like we do here:

private static (ActivitySource ActivitySource, Meter Meter) CreateTelemetry()
{
const string telemetrySchemaUrl = "https://opentelemetry.io/schemas/1.36.0";
var assembly = typeof(AspNetInstrumentation).Assembly;
var assemblyName = assembly.GetName();
#pragma warning disable IDE0370 // Suppression is unnecessary
var name = assemblyName.Name!;
#pragma warning restore IDE0370 // Suppression is unnecessary
var version = assembly.GetPackageVersion();
var activitySourceOptions = new ActivitySourceOptions(name)
{
Version = version,
TelemetrySchemaUrl = telemetrySchemaUrl,
};
var meterOptions = new MeterOptions(name)
{
Version = version,
TelemetrySchemaUrl = telemetrySchemaUrl,
};
return (new ActivitySource(activitySourceOptions), new Meter(meterOptions));
}

@MattKotsenas MattKotsenas force-pushed the feature/kusto-instrumentor branch from 63134a9 to 3a97216 Compare April 7, 2026 04:51
/// </summary>
internal static class KustoActivitySourceHelper
{
public const string DbSystem = "azure.kusto";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ConiderDbSystemNameValue / KustoDbSystemNameValue / AzureKustoDbSystemNameValue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:instrumentation.kusto Things related to OpenTelemetry.Instrumentation.Kusto dependencies Pull requests that update a dependency file infra Infra work - CI/CD, code coverage, linters keep-open Prevents issues and pull requests being closed as stale perf Performance related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature request] Instrumentor for Azure Kusto

6 participants