XIP-83: Mutable subscription streams with liveness#139
Open
tylerhawkes wants to merge 7 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR adds a new XIP specification document. While documentation-only with no runtime impact, the XIP file is owned by jhaaaa and the author (tylerhawkes) is not the designated code owner, so human review is appropriate. You can customize Macroscope's approvability policy. Learn more. |
Contributor
|
Love this direction. Would be a huge unlock for Herald and also better for all mobile apps. Main concern is that we have an even-more-different code path for the browser and everything else, and that browser issues can go unnoticed. Not a blocker, just an unfortunate side effect. Also means that our client streaming implementation needs to handle both mutable and immutable streams, which makes things harder to maintain. Such is life. |
…multi-identity multiplexing + topology & lifecycle diagrams + browser gRPC-over-WS/WebTransport note
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
June 12, 2026 21:35
087f720 to
25adb20
Compare
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
June 12, 2026 21:59
25adb20 to
5586801
Compare
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
June 15, 2026 20:03
5586801 to
0a71f6c
Compare
…mplete response arms, mutate_id wave correlation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
June 16, 2026 00:17
0a71f6c to
196fb3d
Compare
…vector cursors, OriginatorEnvelope delivery, native-only bidi Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tylerhawkes
added a commit
to xmtp/xmtpd
that referenced
this pull request
Jun 26, 2026
…2020) ## XIP-83 d14n binding — bidirectional `QueryApi.Subscribe` Implements the decentralized (d14n) binding of [XIP-83 mutable subscription streams](xmtp/XIPs#139) on xmtpd. A single long-lived bidirectional stream lets a client **mutate its topic set in place** (add/remove subscriptions) and keep the connection alive with ping/pong — instead of tearing down and reopening a stream every time group membership changes. This is the xmtpd counterpart to the node-go v3 `MlsApi.Subscribe` binding. Same control protocol (`Mutate` / `Started` / `CatchupComplete` / `TopicsLive` / `Ping` / `Pong`); the d14n binding delivers `OriginatorEnvelope`s and uses per-originator vector cursors. ### Design: gated async catch-up - **Single-writer + single-sender actor** per subscription. All topic-set mutation happens on the writer goroutine; all stream sends happen on the sender goroutine (drained via a bounded queue). - **Per-topic gate + pending buffer.** A newly-added topic is caught up by an async fetcher goroutine while already-live topics keep flowing — a slow catch-up never head-of-line-blocks live delivery. - **No missing messages across the switch.** A topic is registered *live* **before** its catch-up snapshot is taken, and delivery is deduped by a monotonic per-originator cursor (`advanceTopicCursors`). Anything that lands during catch-up is either in the snapshot or arrives live; the dedup drops the overlap. - **Sparse vs. filled cursors.** The writer keeps only the originator cursors the client provided (growing them as new originators are seen); the fetcher holds a filled copy purely for the query. Keeps memory bounded at the active-topic ceiling. - **1M active-topic ceiling** (`maxActiveSubscribeTopics`) — ~16–32MB for a consumer that large, deliberately favored over forcing clients into multiple connections (which would invite rate-limiting and wreck some topologies). - **Liveness.** Either peer may `Ping`; the receiver must `Pong` the nonce. No pong within the deadline → the node reaps the vanished peer (e.g. a mobile client the OS suspended behind a proxy that still ACKs the transport). ### Commits 1. `feat(api): add mutableSubscription handle to subscribeWorker` — in-place topic add/remove on the existing subscribe worker, guarded by `topicsMu`. 2. `feat(api): XIP-83 bidirectional Subscribe handler on QueryApi + xmtpv4 protos` — the handler (`subscribe.go`) + regenerated xmtpv4 protos. 3. `test(api): Subscribe handler tests + configurable keepalive interval for tests` — 5 race-tested scenarios + a test-only knob to shorten the keepalive cadence. ### Tests `go test -race ./pkg/api/message` — all green: - `TestSubscribe_CatchUpThenLive` - `TestSubscribe_MutateRemoveStopsDelivery` - `TestSubscribe_HalfCloseHistoryOnlyDrains` - `TestSubscribe_HistoryOnlyOnLiveRejected` - `TestSubscribe_NoPongIsReaped` ### Dependency & merge ordering Built on [xmtp/proto#338](xmtp/proto#338) (`QueryApi.Subscribe`). The committed `.pb.go` here is generated from that branch via `dev/gen/protos`; once #338 merges, regenerating from proto `main` produces byte-identical output. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This was referenced Jul 9, 2026
Contributor
Author
|
Pushed a revision adding server-tagged deliveries and the ordering/seam guarantees: every delivery frame now carries the Spec surface: new server requirements 3 (delivery tagging) and 4 (delivery order and the catch-up seam), old 3–9 renumbered to 5–11 with revisions to 2/5/9/11, client requirement 3 rewritten to the collapsed resume model, the d14n binding extended (per-originator order, vector high-water mark, no-restart re-adds, first-wins dedup), and test cases 15–19. Companion implementation PRs: xmtp/proto#340, xmtp/xmtp-node-go#564, xmtp/xmtpd#2035. |
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
July 9, 2026 08:10
2695ebc to
34619b3
Compare
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
July 9, 2026 08:31
34619b3 to
3311619
Compare
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
July 9, 2026 09:13
3311619 to
b1a5dfb
Compare
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
July 9, 2026 09:32
b1a5dfb to
4475925
Compare
Tag every delivery frame with the catch-up wave that produced it (mutate_id, 0 = live), pin both delivery lanes to cursor order with an exactly-once seam bounded by CatchupComplete, and ack every Mutate with exactly one CatchupComplete. Collapses the client's resume state to one live high-water mark per stream plus a transient progress mark per in-flight wave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tylerhawkes
force-pushed
the
tyler/xip-83-mutable-subscription-streams
branch
from
July 9, 2026 17:07
4475925 to
8162dc3
Compare
tylerhawkes
added a commit
to xmtp/xmtp-node-go
that referenced
this pull request
Jul 10, 2026
…and enforce ordering guarantees (#564) Implements XIP-83 server-tagged deliveries on the v3 `Subscribe` handler. Spec revision: xmtp/XIPs#139; protos: xmtp/proto#340. ## Protocol behavior - Every `Messages` frame is stamped with the wave that produced it (`mutate_id`, `0` = live); a frame never mixes lanes or waves. - Live frames deliver in ascending id order per message kind, stream-wide; a wave's replay is one merged id-ordered pass across all its topics, per kind. - The seam: no live frame for a wave's topic before that wave's `CatchupComplete`; live for other topics keeps flowing; messages are delivered exactly once across the seam. - A `Mutate` with adds must carry a nonzero `mutate_id` (`INVALID_ARGUMENT` otherwise); every `Mutate` is acked by exactly one `CatchupComplete` (immediate when no wave starts). ## Implementation - Chunked per-topic catch-up is replaced by a per-wave merged keyset scan per kind (`QueryGroupMessagesWaveScan` / `QueryWelcomeMessagesWaveScan`: unnest-join per-topic floors, global scan cursor, ceiling pinned at wave start, `ORDER BY id LIMIT page`). - Live arrivals for a wave's topics are gated into per-topic pending buffers and folded into the wave at completion — sorted per kind, tagged, deduped against what the scan delivered — then `TopicsLive`, then `CatchupComplete`, then the gates open. - Welcome wave scans advance the cursor from raw rows, so a row with an unknown `message_type` can no longer silently truncate a wave's replay. - `Mutate` processing parses and validates all adds before applying removes. Depends on xmtp/proto#340 — the vendored generated code here matches that branch and regenerates identically from proto@main once it lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
tylerhawkes
added a commit
to xmtp/xmtpd
that referenced
this pull request
Jul 14, 2026
…enforce ordering guarantees (#2035) Implements XIP-83 server-tagged deliveries on the d14n `QueryApi.Subscribe` handler. Spec revision: xmtp/XIPs#139; protos: xmtp/proto#340. ## Protocol behavior - Every `Envelopes` frame is stamped with the wave that produced it (`mutate_id`, `0` = live); a frame never mixes lanes or waves. - Live frames deliver each originator's envelopes in ascending `originator_sequence_id`, stream-wide; a wave's replay does the same across the wave's topics. - The seam: no live frame for a wave's topic before that wave's `CatchupComplete`; live for other topics keeps flowing; envelopes are delivered exactly once across the seam. - A `Mutate` with adds must carry a nonzero `mutate_id` (`INVALID_ARGUMENT` otherwise); every `Mutate` is acked by exactly one `CatchupComplete`. ## Implementation - Per-topic chunked catch-up is replaced by one merged keyset scan (`SelectGatewayEnvelopesWaveScan`: per-(topic, originator) floors, row-value `(originator, sequence)` keyset progression, per-originator ceilings from the new `SelectOriginatorCeilings` pinned at wave start). - Live arrivals for a wave's topics are gated and folded into the wave at completion (sorted per originator, tagged, deduped via the topic cursor), then `TopicsLive`, then `CatchupComplete`, then the gates open. - The scan position advances from raw rows; page fetches retry with backoff and send only after a fully successful fetch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft XIP, opened for circulation/discussion.
Summary
Defines a single bidirectional subscription RPC (
Subscribe) on the MLS API. A client opens one long-lived stream and mutates its topic set in place (add/remove deltas) instead of tearing down and reopening on every membership change; the server delivers messages plus a periodic liveness heartbeat (StatusUpdate{ WAITING }) so clients can detect silent stream death that transport keepalives miss — a terminating L7 proxy answers HTTP/2 pings at the edge while the backend subscription is gone. One connection can carry the union of many topics, the enabling primitive for multi-tenant agent gateways.Compatibility
Additive and backward-compatible: existing
SubscribeGroupMessages/SubscribeWelcomeMessagesare untouched, and WASM/browser clients (no bidirectional gRPC) stay on them. A client callingSubscribeagainst a node that lacks it getsUNIMPLEMENTEDand falls back.Status
Draft, for discussion. The client-side liveness floor — a
WatchdogStreamcombinator that reconnects a stale subscription from its persisted cursor — is already implemented in libxmtp against the existing server-streaming subscriptions (xmtp/libxmtp#3718). TheSubscribeRPC + node heartbeat handler standardized here are the remaining protocol work.Note
Add XIP-83 specification for mutable subscription streams with liveness
Adds xip-83-mutable-subscription-streams.md, a protocol specification for bidirectional mutable subscription streams. The spec covers ping/pong liveness checks, delivery tagging via
mutate_id,TopicsLiveandCatchupCompletemarkers, client/server requirements, decentralized binding, backward compatibility, test cases, and security considerations.Macroscope summarized 8162dc3.