Skip to content

[chore][processor/dynamic_sampling] add concurrent ConsumeTraces benchmarks - #50047

Merged
ChrsMark merged 2 commits into
open-telemetry:mainfrom
MikeGoldsmith:mike/dynamicsampling-parallel-benchmarks
Aug 5, 2026
Merged

[chore][processor/dynamic_sampling] add concurrent ConsumeTraces benchmarks#50047
ChrsMark merged 2 commits into
open-telemetry:mainfrom
MikeGoldsmith:mike/dynamicsampling-parallel-benchmarks

Conversation

@MikeGoldsmith

@MikeGoldsmith MikeGoldsmith commented Aug 4, 2026

Copy link
Copy Markdown
Member

Description

All existing benchmarks drive ConsumeTraces from a single goroutine, but the collector calls it concurrently from receivers and pipelines, and the processor serializes most of its work behind one mutex. This adds concurrent benchmarks so multi-core scaling is measured rather than assumed, and gives the planned lock-restructuring work a baseline and acceptance test.

BenchmarkConsumeTraces_Parallel covers the steady-state paths (new traces, appends to pending traces, decision-cache hits for both outcomes) and BenchmarkConsumeTraces_ParallelMixed runs ingest racing timer-driven decisions. Run with -cpu=1,4,8,16. The file header documents a -mutexprofile recipe for attributing lock wait.

Measured today (spans/sec, mean of 3, Apple M4 Pro):

path 1 cpu 4 cpu 8 cpu 16 cpu
new_traces 1.35M 2.56M 2.58M 2.61M
append_to_pending 2.28M 3.40M 3.55M 3.57M
cache_hit_sampled 2.25M 2.81M 2.85M 2.81M
cache_hit_dropped 30.1M 23.8M 23.1M 22.6M
mixed ingest + decisions 1.16M 1.79M 1.63M 1.42M

The numbers show the ceiling clearly. Every path stops scaling at 4 cores, the cheapest path (cache-hit drop) gets slower under contention, and the mixed workload regresses past 4 cores once decision timers compete for the same lock.

Link to tracking issue

Refs #49311

Testing

  • This PR is benchmarks; the table above is the output of running them.
  • Full test suite and lint pass.

Documentation

Run instructions and interpretation guidance are in the file header. Test-only change, no changelog entry.

Authorship

  • I, a human, wrote this pull request description myself.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Aug 4, 2026

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-08-05 10:38 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@MikeGoldsmith
MikeGoldsmith marked this pull request as ready for review August 5, 2026 09:35
@MikeGoldsmith
MikeGoldsmith requested a review from a team as a code owner August 5, 2026 09:35
@MikeGoldsmith
MikeGoldsmith requested a review from dmitryax August 5, 2026 09:35
@github-actions
github-actions Bot requested a review from VinozzZ August 5, 2026 09:35
@ChrsMark
ChrsMark merged commit 7d68d60 into open-telemetry:main Aug 5, 2026
139 checks passed
@MikeGoldsmith
MikeGoldsmith deleted the mike/dynamicsampling-parallel-benchmarks branch August 5, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants