fix(relay): gzip large PS responses so big exports survive the relay tunnel (BUI-591)#166
Merged
Merged
Conversation
…tunnel (BUI-591) (#164) ## Problem Large exports (e.g. `chatgpt.conversations`) are big JSON streamed **uncompressed** over the relay's WebSocket tunnel. The longer the transfer, the wider the window for a mid-stream drop, which the SDK/client sees as a truncated read (`UND_ERR_RES_CONTENT_LENGTH_MISMATCH`). Small scopes (e.g. `instagram.profile`) finish inside the reliable window and already work. Full analysis in **BUI-591** — this is fix #1 of the prioritized minimal set. ## Change gzip the response body in the relay's HTTP serialization when: the client sent `Accept-Encoding: gzip` (undici + browsers do, and auto-decompress — transparent to builders), the body is large enough to be worth it (`>= 1KB`), and it isn't already encoded. JSON typically shrinks 5–10×, bringing large exports into the same reliable window as small scopes (fewer WS frames, faster transfer, less browser memory). Implementation: - `buildHttpResponse` stays **sync** (existing callers/tests unaffected) via an optional `bodyOverride` + `contentEncoding`; the async gzip runs in `handleRelayHttpRequest`. - Sets `content-encoding: gzip` and a `content-length` matching the compressed body. ## Verification - `npm run build` clean (tsc --build, all packages). - `packages/lite` tests: **114 passed**, incl. the existing binary-safety regression and a new test for the override / content-encoding path. ## Notes - Complements the existing 16KB response chunking (compression cuts total bytes/frames; chunking bounds frame size). - Addresses the **large-payload truncation** axis of BUI-591. The separate **availability** axis (browser PS flapping / TLS resets when the tab isn't tunneled) is out of scope here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codex ReviewFindings
Verification
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-lands the BUI-591 gzip fix on
dev— the branch the deployed surfaces actually consume (unity-surfaces vendors compileddist/fromdevfor app-dev.vana.org). The original PR #164 merged intomain, which has diverged fromdev, so it could not reach app-dev; #165 reverts it frommain.Clean cherry-pick of bfa9225 (
relay.tsis byte-identical on both branches at the base). Verified ondev:npm run buildclean, fullpackages/litesuite passes (121 tests, incl. the two BUI-591 relay cases).What it does
Large exports (e.g.
chatgpt.conversations) are big JSON streamed over the relay's WebSocket tunnel; an uncompressed body inflates transfer time and the window for a mid-stream drop (truncated read →UND_ERR_RES_CONTENT_LENGTH_MISMATCH). When the client sendsAccept-Encoding: gzip(undici does, automatically, and auto-decompresses) and the body is large enough, the relay gzips the response and setscontent-encoding: gzip. JSON shrinks ~5–10× → transfer finishes inside the reliable window small scopes already enjoy.Correctness (verified)
res.json()and the resilientarrayBuffer()paths).content-lengthbefore serializing so the gzip response can't carry two conflicting lengths (Codex review catch from fix(relay): gzip large PS responses so big exports survive the relay tunnel (BUI-591) #164).Guards
Only fires when the client accepts gzip, skips already-encoded bodies, no-ops where
CompressionStreamis absent, ≥1KB threshold.Closes BUI-591.