Drop too large WAL events - #46
Open
udnay wants to merge 1 commit into
Open
Conversation
…s each row change to Google Pub/Sub. Google Pub/Sub has a hard 10 MiB per-message limit. When wal-listener tries to publish a marshaled event bigger than that, the Google Cloud Pub/Sub Go client's bundler rejects it with the error "item size exceeds bundle byte limit" and returns it to our result handler, which logs a WARN. In production on wal-listener-hot-sauce, a specific app's row has grown so large that every UPDATE to it produces a >10 MiB serialized event. Because logical-decoding UPDATE events include both old and new tuple values (Data and DataOld in publisher/event.go), a 6 MiB JSONB column produces a ~12 MiB event envelope. These drops spam the log every second or so and create ambiguity about whether they're being silently retried or lost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Google Pub/Sub has a hard 10 MiB per-message limit. When wal-listener tries to publish a marshaled event bigger than that, the Google
Cloud Pub/Sub Go client's bundler rejects it with the error "item size exceeds bundle byte limit" and returns it to our result handler, which logs a WARN.
In production on wal-listener-hot-sauce, a specific app's row has grown so large that every UPDATE to it produces a >10 MiB serialized event. Because logical-decoding UPDATE events include both old and new tuple values (Data and DataOld in
publisher/event.go), a 6 MiB JSONB column produces a ~12 MiB event envelope. These drops spam the log every second or so and create ambiguity about whether they're being silently retried or lost.
Note
Medium Risk
Adds intentional data-loss behavior for oversized Pub/Sub messages and changes how publish results are interpreted, so mis-sizing or unexpected payload growth could silently drop events while still advancing the replication slot.
Overview
Prevents Google Pub/Sub client bundler failures by adding a size guard in
GooglePubSubPublisher.Publish: events that marshal abovemaxPubSubMessageBytesare dropped with a warning and return a nil-errorPublishResult.Refactors listener result processing into
Listener.handlePublishResult, and updates it to treat a nil-errorPublishResultas a successful no-op (increment publish metrics, return the event to the pool) while still advancinglatestWalStartfromXLogData/keepalives so WAL acks progress. Adds unit tests covering oversized-drop behavior inpublisher/pubsub_test.goand the listener’s handling of nil-error and nil-result publish outcomes.Reviewed by Cursor Bugbot for commit cc32d77. Bugbot is set up for automated code reviews on this repo. Configure here.