Skip to content

[FEATURE] Store the source partition key as decomposed columns in the JDBC source coordination store #6982

Description

@lawofcycles

Is your feature request related to a problem? Please describe.
In the JDBC source coordination store (#6757), source_partition_key is a single opaque VARCHAR(512) column that forms the composite primary key together with source_identifier. Source plugins encode structured information into this key (for example bucket, object key, and scan range for S3-based sources), but the store flattens it into one string.

As a result:

  • Operational queries against the coordination table (e.g. "show all partitions for bucket X") require LIKE matching on the encoded string
  • The whole key must fit in one column, which is capped at VARCHAR(512) by MySQL's 3072-byte composite index limit
  • Individual key components cannot be indexed

Describe the solution you'd like
Store the partition key components in separate columns in the JDBC store schema, keeping (source_identifier, source_partition_key) as the logical identity. For example, nullable component columns (partition_key_part_1 .. partition_key_part_n), or a JSON column on databases that support it.

Design considerations
Today each source plugin invents its own key format, and the SourceCoordinationStore interface in data-prepper-api only sees an opaque string: S3-based sources use bucket|key-style segments, DynamoDB uses export ARNs and shard IDs, Iceberg uses file paths and ranges. There is no cross-source convention, not even for the separator character.

So decomposing the key requires one of:

  1. A documented key convention that all source plugins follow (a cross-plugin contract, and existing keys in the wild would not conform), or
  2. An extension of the SourceCoordinationStore interface so sources pass structured components explicitly (an API change affecting every store implementation, including DynamoDB and in-memory)

Additional context
The single-column schema shipped in #6757 keeps working as is. Adding nullable component columns later is backward compatible, so deferring this does not create a migration problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Unplanned

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions