Add gRPC keepalive max_connection_age settings on OTel sources (#6745)#6947
Open
srikanthpadakanti wants to merge 2 commits into
Open
Add gRPC keepalive max_connection_age settings on OTel sources (#6745)#6947srikanthpadakanti wants to merge 2 commits into
srikanthpadakanti wants to merge 2 commits into
Conversation
…earch-project#6745) Expose `max_connection_age` and `connection_drain_duration` on every Data Prepper source that exposes a gRPC server (otlp, otel-logs, otel-trace, otel-metrics). Both settings are optional `Duration` fields on the existing per-source configuration. When set, they are applied to the Armeria `ServerBuilder` via `maxConnectionAge(Duration)` and `connectionDrainDuration(Duration)`. When unset they preserve current behavior (no upper bound on connection age). This addresses the horizontal-scaling problem where long-lived gRPC channels never re-resolve DNS to discover new replicas. Configuring `max_connection_age` forces clients to reconnect periodically and pick up new endpoints. Implementation details: - New `Duration` fields with `@DurationMin` / `@DurationMax` validation on `OTLPSourceConfig`, `OTelLogsSourceConfig`, `OTelTraceSourceConfig`, `OTelMetricsSourceConfig`. - `ServerConfiguration` in `http-common` carries the two new fields so `otlp-source` and `otel-metrics-source` pick them up through the shared `CreateServer.createGRPCServer(...)` path. `otel-logs-source` and `otel-trace-source` apply them directly to their inline `ServerBuilder`. - `hibernate-validator` is promoted to the version catalog (`libs.hibernate.validator`) so all four source modules share a single pinned version, replacing the inline pin previously in `otlp-source`. Resolves opensearch-project#6745 Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
srikanthpadakanti
requested review from
KarstenSchnitter,
Zhangxunmt,
dinujoh,
divbok,
dlvenable,
graytaylor0,
kkondaka,
oeyh,
san81,
sb2k16 and
srikanthjg
as code owners
June 30, 2026 20:40
Collaborator
Author
|
@dlvenable please review this |
kkondaka
reviewed
Jul 14, 2026
| @JsonProperty("max_connection_age") | ||
| @DurationMin(seconds = 1) | ||
| @DurationMax(hours = 24) | ||
| private Duration maxConnectionAge; |
Collaborator
There was a problem hiding this comment.
Where is the default value set?
Collaborator
Author
There was a problem hiding this comment.
@kkondaka No default on purpose. It's optional, stays null when unset, and we only apply it when non-null, so omitting it keeps Armeria's default and today's behavior. Can add an explicit default if you'd prefer.
| @JsonProperty("connection_drain_duration") | ||
| @DurationMin(millis = 0) | ||
| @DurationMax(hours = 1) | ||
| private Duration connectionDrainDuration; |
Collaborator
There was a problem hiding this comment.
Where is the default value set?
Collaborator
Author
There was a problem hiding this comment.
Same here, no default. Null when unset and only kicks in with max_connection_age, so it's a no-op unless configured
kkondaka
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds two optional gRPC server keepalive settings on every Data Prepper source that exposes a gRPC server:
max_connection_ageandconnection_drain_duration. These map directly to Armeria'sServerBuilder.maxConnectionAge(Duration)andServerBuilder.connectionDrainDuration(Duration), which are Armeria's implementations of gRPC's nativeMAX_CONNECTION_AGEandMAX_CONNECTION_AGE_GRACE.Use case (from the issue): horizontally scaled Data Prepper deployments. gRPC clients do not re-resolve DNS once a channel is established, so new replicas never receive traffic until clients reconnect. Setting
max_connection_ageon the server side forces clients to reconnect periodically and pick up new endpoints.Sources covered
All four gRPC-server-exposing sources:
otlp-sourceotel-trace-sourceotel-logs-sourceotel-metrics-sourceConfiguration example
Issues Resolved
Resolves #6745
#6745
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.