Is your feature request related to a problem? Please describe.
In the JDBC source coordination store (#6757), source_partition_key is a single opaque VARCHAR(512) column that forms the composite primary key together with source_identifier. Source plugins encode structured information into this key (for example bucket, object key, and scan range for S3-based sources), but the store flattens it into one string.
As a result:
- Operational queries against the coordination table (e.g. "show all partitions for bucket X") require
LIKE matching on the encoded string
- The whole key must fit in one column, which is capped at
VARCHAR(512) by MySQL's 3072-byte composite index limit
- Individual key components cannot be indexed
Describe the solution you'd like
Store the partition key components in separate columns in the JDBC store schema, keeping (source_identifier, source_partition_key) as the logical identity. For example, nullable component columns (partition_key_part_1 .. partition_key_part_n), or a JSON column on databases that support it.
Design considerations
Today each source plugin invents its own key format, and the SourceCoordinationStore interface in data-prepper-api only sees an opaque string: S3-based sources use bucket|key-style segments, DynamoDB uses export ARNs and shard IDs, Iceberg uses file paths and ranges. There is no cross-source convention, not even for the separator character.
So decomposing the key requires one of:
- A documented key convention that all source plugins follow (a cross-plugin contract, and existing keys in the wild would not conform), or
- An extension of the
SourceCoordinationStore interface so sources pass structured components explicitly (an API change affecting every store implementation, including DynamoDB and in-memory)
Additional context
The single-column schema shipped in #6757 keeps working as is. Adding nullable component columns later is backward compatible, so deferring this does not create a migration problem.
Is your feature request related to a problem? Please describe.
In the JDBC source coordination store (#6757),
source_partition_keyis a single opaqueVARCHAR(512)column that forms the composite primary key together withsource_identifier. Source plugins encode structured information into this key (for example bucket, object key, and scan range for S3-based sources), but the store flattens it into one string.As a result:
LIKEmatching on the encoded stringVARCHAR(512)by MySQL's 3072-byte composite index limitDescribe the solution you'd like
Store the partition key components in separate columns in the JDBC store schema, keeping
(source_identifier, source_partition_key)as the logical identity. For example, nullable component columns (partition_key_part_1..partition_key_part_n), or a JSON column on databases that support it.Design considerations
Today each source plugin invents its own key format, and the
SourceCoordinationStoreinterface in data-prepper-api only sees an opaque string: S3-based sources usebucket|key-style segments, DynamoDB uses export ARNs and shard IDs, Iceberg uses file paths and ranges. There is no cross-source convention, not even for the separator character.So decomposing the key requires one of:
SourceCoordinationStoreinterface so sources pass structured components explicitly (an API change affecting every store implementation, including DynamoDB and in-memory)Additional context
The single-column schema shipped in #6757 keeps working as is. Adding nullable component columns later is backward compatible, so deferring this does not create a migration problem.