Drop transactions from a foreign origin take 2 - #42
Merged
Conversation
The databases we listen to at Gadget see really huge transactions when we do shard moves. Shard moves use logical replication to move data from one database to another, and logical replication does a bigass COPY statement for each table at the start of replication to move the initial state of the table from the source database to the destination. These create huge transactions in the WAL which the wal-listener needs to process. We don't really care about this data downstream -- we in fact don't want it at all, as it is just a replay of data on a new shard that we've already seen on the old shard. Luckily, postgres has a property in the replication protocol that allows us to tell postgres that we don't actually want these transactions streamed to us in the wal-listener: the `ORIGIN` option of a subscription! When you create a subscription, you can specify that you only want messages with no origin, which means only messages that originate from the instance the subscription is on, instead of any upstream instances: ``` primary1=# CREATE SUBSCRIPTION sub_pri1_pri3 primary1-# CONNECTION 'dbname=foo host=primary3 user=repuser' primary1-# PUBLICATION pub_pri3 WITH (origin = NONE); ``` This matches what we want when using the wal-listener at Gadget: we want transactions from a data plane database for envs that currently live on that database. But, when an env is moving to that database, the transactions will have an origin of the upstream shard move source, and we don't want that data, as we're still actually serving the data from that source, and listening to it too. Sadly, this `ORIGIN` option is only supported on PG 16 and newer, and we were on PG 15 at the time, so we couldn't use it. This PR switches that around, as now we're on PG 17! Horray At the time though, we were persistent, and tried to make due with a different, lower level feature of the postgres logical replication protocol, where each incoming message includes an `origin` property that describes where it came from. We hoped we could inspect this property and quickly drop messages that had origins we didn't care about as well. We added the `dropForeignOrigin` configuration flag to the wal-listener to set this flag in #31. Even more sadly, we learned the hard way that while the `origin` property of was present in the older postgres versions, and valid in the protocol, it was just never set at all. Sad. So, this PR drops support for the old, busted, message-inspection based way of origin filtering, in favour of the better peforming, server-side filtering using the ORIGIN property of the subscription.
airhorns
changed the title
airhorns
commented
Dec 3, 2025
| // Origin message format: | ||
| // Int64 - LSN of the commit on the origin server | ||
| // String - Name of the origin | ||
| _ = p.readInt64() // Skip the LSN, we only need the name |
Author
There was a problem hiding this comment.
this doesn't really matter, but claude noticed it was wrong so we fixed it
airhorns
commented
Dec 3, 2025
| func (w *WalTransaction) ShouldDropMessage() bool { | ||
| return w.dropForeignOrigin && w.origin != "" | ||
| } | ||
|
|
Author
There was a problem hiding this comment.
dont need any of this anymore, postgres does it for us
airhorns
commented
Dec 3, 2025
| // Requires PostgreSQL 16+. | ||
| if l.cfg.Listener.DropForeignOrigin { | ||
| pluginArgs = append(pluginArgs, "origin 'none'") | ||
| } |
Author
There was a problem hiding this comment.
this is the entire implementation now :)
scott-rc
approved these changes
Dec 3, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The databases we listen to at Gadget see really huge transactions when we do shard moves. Shard moves use logical replication to move data from one database to another, and logical replication does a bigass COPY statement for each table at the start of replication to move the initial state of the table from the source database to the destination.
These create huge transactions in the WAL which the wal-listener needs to process. We don't really care about this data downstream -- we in fact don't want it at all, as it is just a replay of data on a new shard that we've already seen on the old shard. Luckily, postgres has a property in the replication protocol that allows us to tell postgres that we don't actually want these transactions streamed to us in the wal-listener: the
ORIGINoption of a subscription!When you create a subscription, you can specify that you only want messages with no origin, which means only messages that originate from the instance the subscription is on, instead of any upstream instances:
This matches what we want when using the wal-listener at Gadget: we want transactions from a data plane database for envs that currently live on that database. But, when an env is moving to that database, the transactions will have an origin of the upstream shard move source, and we don't want that data, as we're still actually serving the data from that source, and listening to it too.
Sadly, this
ORIGINoption is only supported on PG 16 and newer, and we were on PG 15 at the time, so we couldn't use it. This PR switches that around, as now we're on PG 17! HorrayAt the time though, we were persistent, and tried to make due with a different, lower level feature of the postgres logical replication protocol, where each incoming message includes an
originproperty that describes where it came from. We hoped we could inspect this property and quickly drop messages that had origins we didn't care about as well. We added thedropForeignOriginconfiguration flag to the wal-listener to set this flag in #31. Even more sadly, we learned the hard way that while theoriginproperty of was present in the older postgres versions, and valid in the protocol, it was just never set at all. Sad.So, this PR drops support for the old, busted, message-inspection based way of origin filtering, in favour of the better peforming, server-side filtering using the ORIGIN property of the subscription.