Add otel.sdk.processor.log.processed metric by cijothomas · Pull Request #7486 · open-telemetry/opentelemetry-dotnet

cijothomas · 2026-07-03T14:34:29Z

Towards #3880

Adds the otel.sdk.processor.log.processed counter metric to BatchLogRecordExportProcessor and SimpleLogRecordExportProcessor, following the OTel SDK self-observability semantic conventions.

This is the first SDK self-observability metric in the .NET SDK. More metrics (queue size, exporter counters, etc.) will follow in subsequent PRs.

Design

Meter: otel.sdk.experimental - requires explicit opt-in via .AddMeter("otel.sdk.experimental"). Without this, Counter.Add() is a no-op with zero overhead.
Instrument: Counter<long> named otel.sdk.processor.log.processed
Attributes:
- otel.component.type - batching_log_processor or simple_log_processor
- otel.component.name - unique per instance (e.g. batching_log_processor/0)
- error.type (conditional) - queue_full when buffer rejects, already_shutdown when processor has been shut down

Opt-in example

var meterProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter("otel.sdk.experimental")
    .AddOtlpExporter()
    .Build();

Why .AddMeter() instead of env variable or compile-time flag?

Metrics in .NET are inherently opt-in: without a MeterProvider subscribing to the meter, Counter.Add() is a no-op at near-zero cost. The .AddMeter("otel.sdk.experimental") call provides natural gating without requiring additional mechanisms like env variables or feature flags.

Implement the otel.sdk.processor.log.processed counter metric for BatchLogRecordExportProcessor and SimpleLogRecordExportProcessor. The metric is emitted on meter 'otel.sdk.experimental' which requires explicit opt-in via .AddMeter("otel.sdk.experimental") — zero overhead without a listener. Attributes: - otel.component.type: batching_log_processor | simple_log_processor - otel.component.name: unique instance identifier (e.g. batching_log_processor/0) - error.type (conditional): queue_full | already_shutdown Design: - SdkSelfObservability internal helper owns the Meter and Counter - Pre-allocated KeyValuePair[] tag arrays (no per-call allocation) - Batch processor uses TryExport return value to classify success vs queue_full - Both processors detect post-shutdown state via OnShutdown override

codecov · 2026-07-03T14:46:09Z

Codecov Report

❌ Patch coverage is 95.16129% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.23%. Comparing base (8592991) to head (2a53a4a).
⚠️ Report is 26 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/OpenTelemetry/Internal/MeterFactory.cs	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7486      +/-   ##
==========================================
+ Coverage   90.21%   90.23%   +0.01%     
==========================================
  Files         287      289       +2     
  Lines       15560    15620      +60     
==========================================
+ Hits        14038    14094      +56     
- Misses       1522     1526       +4

Flag	Coverage Δ
unittests-Project-Experimental	`90.26% <95.16%> (+<0.01%)`	⬆️
unittests-Project-Stable	`90.21% <95.16%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/OpenTelemetry/Internal/SdkSelfObservability.cs	`100.00% <100.00%> (ø)`
...ry/Logs/Processor/BatchLogRecordExportProcessor.cs	`100.00% <100.00%> (+11.11%)`	⬆️
...y/Logs/Processor/SimpleLogRecordExportProcessor.cs	`100.00% <100.00%> (ø)`
src/OpenTelemetry/Internal/MeterFactory.cs	`80.00% <80.00%> (ø)`

... and 8 files with indirect coverage changes

The test was flaky on Windows ARM / net462 because with maxQueueSize=1 and maxExportBatchSize=1, the worker thread could drain the queue faster than items were being emitted, resulting in no drops. Use a blocking DelegatingExporter to hold the worker in Export() while filling the queue.

martincostello · 2026-07-03T15:49:22Z

                break;
        }
+
+        // TODO: Consider switching to an ObservableCounter that piggybacks on


Issue for tracking?

Will file a tracking issue for this.

Cool - just add a link to the TODO once created.

- Copy MeterFactory from contrib for auto-versioning - Make Meter internal (removes SA1202 suppression) - Share base tags between success/error arrays - Single Counter.Add() call with local tag assignment - Shorten README overhead note

martincostello · 2026-07-04T07:30:24Z

 /// </summary>
 public class BatchLogRecordExportProcessor : BatchExportProcessor<LogRecord>
 {
+    private static int instanceCounter = -1;


While unlikely there'd ever be that many created in the lifetime of a process, maybe use long to give more headroom against overflow?

martincostello · 2026-07-04T07:30:42Z

                break;
        }
+
+        // TODO: Consider switching to an ObservableCounter that piggybacks on


Cool - just add a link to the TODO once created.

github-actions · 2026-07-12T04:19:25Z

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

Kielek · 2026-07-13T06:30:53Z

@cijothomas, I asked Codex /GPT 5.6 Sol. Briefly checked findings and it looks valid to me:

Following comment is AI driven

Reviewed commit: 2a53a4a

1. Count batch success at exporter submission, not queue admission

Priority: P1

BatchLogRecordExportProcessor.cs:108
increments the success counter immediately after TryExport enqueues the log
record.

The semantic convention says that, for simple and batching processors, a log
record is considered processed when it has been submitted to the exporter, not
when the corresponding export call finishes. For the batching processor,
submission happens later on the worker thread. The current implementation can
therefore report records as successfully processed while they are still queued.
If draining does not complete, some of those records may never be submitted to
the exporter.

The successful increment should happen at the worker/export boundary. Queue
rejection failures can continue to be recorded where enqueueing is attempted.
The BatchProcessor_SuccessfulEnqueue test should also verify that success is
not recorded before the exporter receives the record.

Specification:
https://opentelemetry.io/docs/specs/semconv/otel/sdk-metrics/#metric-otelsdkprocessorlogprocessed

2. The simple processor reports exported records as `already_shutdown`

Priority: P1

SimpleLogRecordExportProcessor.cs:38-42
calls base.OnEnd(data) before checking isShutdown.

SimpleExportProcessor does not prevent OnEnd from invoking
Exporter.Export after Shutdown() has been called. As a result, a record
emitted after shutdown is still submitted to the exporter, but the new metric
labels it with error.type=already_shutdown. The README describes that record
as lost, although it was exported.

The processor should either reject records after shutdown before invoking the
exporter, or report the successful submission that actually occurred. The
shutdown test should retain the exporter's collection and assert whether the
post-shutdown record was exported.

The semantic convention reserves already_shutdown for records dropped because
the processor has already been shut down:
https://opentelemetry.io/docs/specs/semconv/otel/sdk-metrics/#metric-otelsdkprocessorlogprocessed

3. Shutdown races can mislabel successfully queued batch records

Priority: P2

BatchLogRecordExportProcessor.cs:95-108
reads isShutdown after attempting to enqueue the record.

For example, the following interleaving is possible:

OnEnd successfully enqueues a record.
A concurrent Shutdown() sets isShutdown and includes the record in its
drain target.
OnEnd observes isShutdown and records already_shutdown.
The worker submits the record to the exporter during shutdown draining.

This reports a successfully processed record as a shutdown failure. Acceptance
and shutdown need an atomic outcome, rather than classifying the result using a
separate volatile flag after enqueueing.

4. Add the required SDK CHANGELOG entry

Priority: P2

This is a behavioral addition to the OpenTelemetry component, but the changed
files do not include src/OpenTelemetry/CHANGELOG.md. Repository policy requires
an entry at the end of that file's Unreleased section, using PR #7486 in the
link.

martincostello · 2026-07-16T07:36:34Z

        {
            using (var batch = new Batch<T>(this.CircularBuffer, this.MaxExportBatchSize))
            {
+                this.exportStarted?.Invoke(batch.Count);


Should we defend against this callback throwing so that Export() is still called?

martincostello · 2026-07-16T07:37:40Z

+        SpinWait spinner = default;
+        while (Volatile.Read(ref this.activeOnEndCount) != 0)
+        {
+            spinner.SpinOnce();
+        }


Should this honour timeoutMilliseconds too?

martincostello · 2026-07-16T07:38:08Z

+        SpinWait spinner = default;
+        while (Volatile.Read(ref this.activeOnEndCount) != 0)


opentelemetry-pr-dashboard · 2026-07-18T13:35:41Z

Pull request dashboard status

Status last refreshed: 2026-07-24 08:59:23 UTC.

Waiting on: Author
Next step: Address or respond to 4 review feedback items:
- Inline threads: 1, 2, 3, 4
- For each item, reply to move the discussion forward, e.g. link to the commit that addresses it, explain why no change is needed, or ask a follow-up question.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected. If you believe this pull request is incorrectly routed as waiting on the author, comment /dashboard route:reviewers to route it from waiting on the author to waiting on reviewers. If the last refreshed time above predates your latest reply or push, the dashboard hasn't processed it yet.

github-actions Bot added the pkg:OpenTelemetry Issues related to OpenTelemetry NuGet package label Jul 3, 2026

Fix CI: replace em-dashes with ASCII, fix markdown table style

89ff78e

cijothomas added this to OTel Clients Self Observability Jul 3, 2026

github-project-automation Bot moved this to Todo in OTel Clients Self Observability Jul 3, 2026

cijothomas moved this from Todo to In progress in OTel Clients Self Observability Jul 3, 2026

cijothomas added 2 commits July 3, 2026 20:24

Add TODO for potential ObservableCounter optimization

48f2b86

cijothomas marked this pull request as ready for review July 3, 2026 15:27

cijothomas requested a review from a team as a code owner July 3, 2026 15:27

martincostello reviewed Jul 3, 2026

View reviewed changes

cijothomas force-pushed the cijothomas/sdk-self-obs-processor-log-processed branch from c6a9a18 to 8c89abe Compare July 4, 2026 01:08

github-actions Bot added the infra Infra work - CI/CD, code coverage, linters label Jul 4, 2026

cijothomas force-pushed the cijothomas/sdk-self-obs-processor-log-processed branch from 8c89abe to 6c5df34 Compare July 4, 2026 01:13

github-actions Bot removed the infra Infra work - CI/CD, code coverage, linters label Jul 4, 2026

cijothomas added 2 commits July 4, 2026 06:51

README: friendlier opt-in wording

c5c90ff

Merge branch 'main' into cijothomas/sdk-self-obs-processor-log-processed

2a53a4a

martincostello approved these changes Jul 4, 2026

View reviewed changes

github-actions Bot added the Stale Issues and pull requests which have been flagged for closing due to inactivity label Jul 12, 2026

Kielek removed the Stale Issues and pull requests which have been flagged for closing due to inactivity label Jul 13, 2026

Fix log processor metric timing and shutdown handling

d79a6af

martincostello reviewed Jul 16, 2026

View reviewed changes

opentelemetry-pr-dashboard Bot mentioned this pull request Jul 17, 2026

Pull Request Dashboard #7539

Open

		SpinWait spinner = default;
		while (Volatile.Read(ref this.activeOnEndCount) != 0)

Uh oh!

Conversation

cijothomas commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design

Opt-in example

Why .AddMeter() instead of env variable or compile-time flag?

Uh oh!

codecov Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

Kielek commented Jul 13, 2026

1. Count batch success at exporter submission, not queue admission

2. The simple processor reports exported records as already_shutdown

3. Shutdown races can mislabel successfully queued batch records

4. Add the required SDK CHANGELOG entry

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opentelemetry-pr-dashboard Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cijothomas commented Jul 3, 2026 •

edited

Loading

codecov Bot commented Jul 3, 2026 •

edited

Loading

2. The simple processor reports exported records as `already_shutdown`

opentelemetry-pr-dashboard Bot commented Jul 18, 2026 •

edited

Loading