Data Saver causes high cardinality for http.server.request.duration

[giliev-longevia]

Describe the bug

Enabling Data Saver with:

GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

causes http.server.request.duration to be aggregated with high-cardinality HTTP attributes, including network.peer.port, url.query, and user_agent.original.

With the flag disabled, the same instrument uses the expected filtered attribute set.

After normal application traffic, the Java SDK reaches the default per-instrument cardinality limit:

[otel.javaagent 2026-07-20 06:48:51:261 +0000] [http-nio-8081-exec-5] WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument http.server.request.duration has exceeded the maximum allowed cardinality (1999).

Data Saver reduces the number of metric names, but for this retained instrument it substantially increases the number of time series.

Steps to reproduce

1. Run a Spring Boot/Tomcat application with grafana-opentelemetry-java.jar v2.29.0.

2. Enable console metric export to inspect the metric datapoints:

   OTEL_METRICS_EXPORTER=otlp,console
   OTEL_METRIC_EXPORT_INTERVAL=10000
   

3. Start once with:

   GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=false
   

4. Send HTTP requests and inspect http.server.request.duration.

5. Restart with:

   GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true
   

6. Send the same traffic and inspect the metric again.

7. With sustained traffic/new client connections, observe the cardinality warning at 1999 series.

Flag disabled

A representative datapoint contains only the bounded HTTP attributes:

name=http.server.request.duration
instrumentationScope=io.opentelemetry.tomcat-10.0
attributes=FilteredAttributes{
  http.request.method=POST,
  http.response.status_code=204,
  http.route=/login/impersonate,
  network.protocol.version=1.1,
  url.scheme=http
}

Across the observed requests, the available/expected dimensions are:

http.request.method
http.response.status_code
http.route
error.type
network.protocol.name
network.protocol.version
url.scheme

Flag enabled

The same instrument contains the full HTTP request attribute set:

name=http.server.request.duration
instrumentationScope=io.opentelemetry.tomcat-10.0
attributes={
  client.address=<redacted>,
  http.request.method=POST,
  http.response.status_code=204,
  http.route=/login/impersonate,
  network.peer.address=<redacted>,
  network.peer.port=50390,
  network.protocol.version=1.1,
  server.address=<redacted>,
  server.port=8081,
  url.path=/login/impersonate,
  url.query=<redacted>,
  url.scheme=http,
  user_agent.original=<redacted>
}

In particular:

• network.peer.port changes for client TCP connections.
• url.query may change per request and may contain sensitive values.
• user_agent.original, client addresses, and peer addresses add further dimensions.

Expected behavior

Data Saver should reduce the set of exported metric instruments without broadening the attribute set of retained instruments.

http.server.request.duration should continue to use the upstream instrumentation's bounded attribute advice:

http.request.method
http.response.status_code
http.route
error.type
network.protocol.name
network.protocol.version
url.scheme

Enabling Data Saver should not cause the application to reach the SDK's 2000-series cardinality limit.

Actual behavior

With Data Saver disabled, the console exporter prints FilteredAttributes containing the bounded HTTP attributes.

With Data Saver enabled, the console exporter prints an unfiltered attribute map containing ephemeral peer ports, query strings, user agents, and addresses. This creates thousands of distinct series for http.server.request.duration and eventually produces:

Instrument http.server.request.duration has exceeded the maximum allowed cardinality (1999).

Environment

JDK: OpenJDK 25.0.2
OS: macOS 26.5.1, aarch64
Framework: Spring Boot 4.1.0, Spring Framework 7.0.8
Server: embedded Tomcat 11.0.22
Grafana Java agent: v2.29.0