Is your feature request related to a problem? Please describe.
The DynamoDB stream source's LeaderScheduler runs shard discovery on a hardcoded 1-minute loop (DEFAULT_LEASE_INTERVAL = Duration.ofMinutes(1)). Since DynamoDB Streams shards roll over continuously, a new child shard is not read until the next discovery pass — making ~1 minute a floor on shard-rotation latency. This shows up as high EndToEndLatency (from the DynamoDB event time) while PipelineLatency stays low: records are processed fast once read, just discovered late.
Describe the solution you'd like
Add an optional shard_discovery_interval under the stream config, defaulting to 1 minute (no behavior change unless set). A shorter interval lowers rotation latency at the cost of more frequent DescribeStream/ListShards calls, which is why the default stays 1 minute.
stream:
start_position: LATEST
shard_discovery_interval: 15s # default: 1m
Additional context
Related to #5208 (shard rotation/latency for the DynamoDB streams source), though that concerns per-worker rotation rather than the leader's discovery cadence. I have a working implementation with tests and would like to contribute it as a PR.
Is your feature request related to a problem? Please describe.
The DynamoDB stream source's
LeaderSchedulerruns shard discovery on a hardcoded 1-minute loop (DEFAULT_LEASE_INTERVAL = Duration.ofMinutes(1)). Since DynamoDB Streams shards roll over continuously, a new child shard is not read until the next discovery pass — making ~1 minute a floor on shard-rotation latency. This shows up as highEndToEndLatency(from the DynamoDB event time) whilePipelineLatencystays low: records are processed fast once read, just discovered late.Describe the solution you'd like
Add an optional
shard_discovery_intervalunder the stream config, defaulting to 1 minute (no behavior change unless set). A shorter interval lowers rotation latency at the cost of more frequentDescribeStream/ListShardscalls, which is why the default stays 1 minute.Additional context
Related to #5208 (shard rotation/latency for the DynamoDB streams source), though that concerns per-worker rotation rather than the leader's discovery cadence. I have a working implementation with tests and would like to contribute it as a PR.