Pass options to wal2json when using kafka integration

[repl-angrez-singh]

I am trying to using build-in kafka support to get events from postgresql to kafka directly using pgstream.
Currently I am able to get events into kafka instance but I am not able to figure out how to get txid in the events.
include-xids: "1" # Enables transaction ID in the JSON payload
But I am unable to figure out how to set this option for pgstream.

Can someone please help?

[eminano]

Hi @repl-angrez-singh !

pgstream doesn't currently support include-xids, as it's not part of the plugin arguments used during the replication initialisation.

I have created an issue to add support for them. We will keep you updated once it's ready, or feel free to open a PR!

Thanks for your feedback! 💟

[repl-angrez-singh]

Thanks @eminano
Do we want to have that as a hard-coded value like others we have currently? Or Do we want to make it configurable (this would be probably require some work)?

[eminano]

Hey @repl-angrez-singh ,

Adding the include-xids to the plugin parameters would mean updating the wal event data as well, since it would include a new field and we'd want to capture it and cascade it downstream. Since it's an additive change (non breaking) we can hardcode it for now, and if in the future we see the need to configure it we can do so (along with other parameters potentially).

[eminano]

The option to add the xids to the replication events has been added in v1.0.1! 🚀

To enable it, you just need to set the replication plugin option in the configuration:

• Env:
  PGSTREAM_POSTGRES_REPLICATION_PLUGIN_INCLUDE_XIDS=true
• Yaml:

replication: # when mode is replication or snapshot_and_replication
      replication_slot: "pgstream_mydatabase_slot"
      plugin:
        include_xids: true # whether to include transaction IDs in the replication stream events

Thank you for your feedback! 💟