Skip to content

Use different strategies for the key and the value in the same topic #573

Description

@vascoferraz

Hi everyone :)

I read the documentation and I did not find any way to use different strategies, that is, TopicNameStrategy, RecordNameStrategy or TopicRecordNameStrategy, for the key and the value in the same topic.

I also tried the following configurations but them failed while deploying:

First attempt - Add two topics entries (one for the value and another for the key)

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalArgumentException: com.purbon.kafka.topology.exceptions.ValidationException: Missing required value.schema.file on schemas for topic xxx-customer-v1

Second attempt: Duplicate the topic entry and set RecordNameStrategy for the value (first entry) and TopicNameStrategy for the key (second entry).

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalStateException: Duplicate key xxx-customer-v1

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions