Skip to content

Drop too large WAL events - #46

Open
udnay wants to merge 1 commit into
mainfrom
yo/drop-large-events
Open

Drop too large WAL events#46
udnay wants to merge 1 commit into
mainfrom
yo/drop-large-events

Conversation

@udnay

@udnay udnay commented Apr 23, 2026

Copy link
Copy Markdown

Google Pub/Sub has a hard 10 MiB per-message limit. When wal-listener tries to publish a marshaled event bigger than that, the Google

Cloud Pub/Sub Go client's bundler rejects it with the error "item size exceeds bundle byte limit" and returns it to our result handler, which logs a WARN.

In production on wal-listener-hot-sauce, a specific app's row has grown so large that every UPDATE to it produces a >10 MiB serialized event. Because logical-decoding UPDATE events include both old and new tuple values (Data and DataOld in
publisher/event.go), a 6 MiB JSONB column produces a ~12 MiB event envelope. These drops spam the log every second or so and create ambiguity about whether they're being silently retried or lost.


Note

Medium Risk
Adds intentional data-loss behavior for oversized Pub/Sub messages and changes how publish results are interpreted, so mis-sizing or unexpected payload growth could silently drop events while still advancing the replication slot.

Overview
Prevents Google Pub/Sub client bundler failures by adding a size guard in GooglePubSubPublisher.Publish: events that marshal above maxPubSubMessageBytes are dropped with a warning and return a nil-error PublishResult.

Refactors listener result processing into Listener.handlePublishResult, and updates it to treat a nil-error PublishResult as a successful no-op (increment publish metrics, return the event to the pool) while still advancing latestWalStart from XLogData/keepalives so WAL acks progress. Adds unit tests covering oversized-drop behavior in publisher/pubsub_test.go and the listener’s handling of nil-error and nil-result publish outcomes.

Reviewed by Cursor Bugbot for commit cc32d77. Bugbot is set up for automated code reviews on this repo. Configure here.

…s each row change to Google Pub/Sub. Google Pub/Sub has a hard 10 MiB per-message limit. When wal-listener tries to publish a marshaled event bigger than that, the Google

  Cloud Pub/Sub Go client's bundler rejects it with the error "item size exceeds bundle byte limit" and returns it to our result handler, which logs a WARN.

  In production on wal-listener-hot-sauce, a specific app's row has grown so large that every UPDATE to it produces a >10 MiB serialized event. Because logical-decoding UPDATE events include both old and new tuple values (Data and DataOld in
  publisher/event.go), a 6 MiB JSONB column produces a ~12 MiB event envelope. These drops spam the log every second or so and create ambiguity about whether they're being silently retried or lost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant