You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[chore][processor/dynamic_sampling] expand documentation for alpha (#50041)
#### Description
Closes out the documentation items on the alpha tracking issue:
- The "How it works" section now states explicitly that samplers only
ever produce rates and that the keep/drop decision is a single mechanism
for every sampler type, the processor's rate-to-threshold conversion
checked against the trace's randomness. A note contrasts this with
dynsampler-go's own samplers (e.g. its `DeterministicSampler` hash
check), which this processor never uses.
- A new "Worked examples" section with two complete configurations:
error retention with an adaptive default rule, and a throughput-bounded
fleet, each with sizing guidance.
Docs only, no behavior change.
#### Link to tracking issue
Refs #49311
#### Testing
The OTTL condition used in the worked examples is covered by the
existing README example compile test. Full test suite and lint pass.
#### Documentation
This PR is documentation.
#### Authorship
- [x] I, a human, wrote this pull request description myself.
Copy file name to clipboardExpand all lines: processor/dynamicsamplingprocessor/README.md
+73-2Lines changed: 73 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,8 +26,11 @@ The Dynamic Sampling Processor performs adaptive tail-based trace sampling using
26
26
-`trace_timeout` elapses since the first span of the trace arrived. This timer is set on first-seen and is never extended by subsequent spans, ensuring a predictable upper bound on buffer occupancy.
27
27
3. After the trigger fires, the processor pauses for `decision_delay` to let in-flight straggler spans land. The same delay applies regardless of which event fired the trigger.
28
28
4. Rules are then evaluated in order against the accumulated trace. The first rule whose conditions all match selects the sampler; once a rule is selected, its sampler's keep/drop decision is final and no later rules are considered. A rule with no conditions is a catch-all.
29
-
5. The matched sampler produces a sample rate (1-in-N).
30
-
6. The keep/drop decision is made deterministically by comparing the sample rate's threshold against the randomness of the trace ID, using the [OTel consistent probability sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) algorithm.
29
+
5. The matched sampler produces a sample rate (1-in-N). Samplers only ever produce rates; no sampler makes a keep/drop decision itself.
30
+
6. The processor converts the rate to a threshold (a rate of N becomes the threshold encoding probability 1/N) and makes the keep/drop decision by comparing that threshold against the trace's randomness (`ot=rv` when present, otherwise derived from the trace ID), using the [OTel consistent probability sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) algorithm.
31
+
32
+
> [!NOTE]
33
+
> The adaptive samplers come from [dynsampler-go](https://github.com/honeycombio/dynsampler-go), which the processor uses **only** to compute rates. dynsampler-go's own samplers can make keep/drop decisions internally (its `DeterministicSampler` applies its own hash-based check, for example), but this processor never uses that path: the rate-to-threshold conversion and the randomness comparison in step 6 are the single decision mechanism for every sampler type, including this processor's `deterministic` type. This is what makes decisions reproducible for a given trace and correctly weighted downstream via `ot=th`.
31
34
7. Sampled traces are forwarded with two annotations on every span:
32
35
-`otelcol.processor.dynamic_sampling.rule`: the name of the matched rule
33
36
- W3C TraceState `ot=th:<hex>`: the threshold encoding the effective sample rate
@@ -293,6 +296,74 @@ sampler:
293
296
294
297
For samplers that accept `key_attributes`, the sampling key for a trace is built by collecting distinct values of each named attribute (across resource and span attributes), sorting them, and joining with the `•` separator. Missing attributes are replaced with `<missing>`.
295
298
299
+
## Worked examples
300
+
301
+
Two complete configurations for the most common deployment shapes.
302
+
303
+
### Error retention with an adaptive default
304
+
305
+
Keep every error trace, and let an adaptive sampler settle the rest at a target percentage per traffic class. This is the right starting shape for most single-instance deployments: errors are never lost, and the default rule adapts as traffic mix shifts.
306
+
307
+
```yaml
308
+
processors:
309
+
dynamic_sampling:
310
+
trace_timeout: 30s
311
+
decision_delay: 2s
312
+
num_traces: 50000
313
+
decision_cache:
314
+
sampled_cache_size: 50000
315
+
non_sampled_cache_size: 100000
316
+
rules:
317
+
# Errors are always kept, evaluated first.
318
+
- name: keep-errors
319
+
conditions:
320
+
- span.status.code == STATUS_CODE_ERROR
321
+
sampler:
322
+
type: always_sample
323
+
# Everything else settles at ~10% per (service, route) class, so
324
+
# low-traffic routes stay visible while hot routes are downsampled.
325
+
- name: default
326
+
sampler:
327
+
type: ema_dynamic
328
+
goal_sampling_percentage: 10
329
+
key_attributes: ["service.name", "http.route"]
330
+
adjustment_interval: 15s
331
+
weight: 0.5
332
+
```
333
+
334
+
Sizing guidance: `num_traces`bounds memory and should cover the number of traces that start within one `trace_timeout` window at peak. The decision caches should be a small multiple of that, since they only store trace IDs and outcomes; undersizing them turns late spans of decided traces back into new partial traces.
335
+
336
+
### Throughput-bounded fleet
337
+
338
+
Cap the spans-per-second this collector emits regardless of incoming volume, useful when the downstream backend is provisioned for a fixed ingest rate. The throughput sampler continuously adjusts per-key rates to hold the total near the goal.
339
+
340
+
```yaml
341
+
processors:
342
+
dynamic_sampling:
343
+
trace_timeout: 30s
344
+
decision_delay: 2s
345
+
num_traces: 100000
346
+
decision_cache:
347
+
sampled_cache_size: 100000
348
+
non_sampled_cache_size: 200000
349
+
# Under sustained overload, shed evicted traces in constant time instead
350
+
# of paying rule evaluation for every one (see Buffer overflow and
351
+
# eviction policy below).
352
+
eviction:
353
+
policy: probabilistic
354
+
sampling_percentage: 10
355
+
rules:
356
+
- name: throughput-cap
357
+
sampler:
358
+
type: windowed_throughput
359
+
goal_throughput_per_sec: 1000
360
+
key_attributes: ["service.name"]
361
+
update_frequency: 1s
362
+
lookback_frequency: 30s
363
+
```
364
+
365
+
The goal is enforced per collector instance: a fleet of N instances emits up to N times the configured throughput, so divide the backend budget by the instance count. Keying by `service.name` means each service's share adapts to its share of total traffic rather than being fixed.
366
+
296
367
## Decision cache
297
368
298
369
When a trace's decision is finalised, the trace ID and outcome are recorded in
0 commit comments