Phase 1: d14n service restructuring by neekolas · Pull Request #328 · xmtp/proto

neekolas · 2026-04-01T20:57:32Z

Summary

Restructures the gRPC service definitions in proto/xmtpv4/ to cleanly separate consumer concerns, as outlined in #327. This is Phase 1 — additive changes with deprecation markers. The only breaking change is moving the unused SubscribeAllEnvelopes RPC to a new NotificationApi service.

New services

QueryApi — Client-to-node queries and subscriptions (QueryEnvelopes, SubscribeTopics, GetInboxIds, GetNewestEnvelope)
PublishApi — Gateway-to-node publishing (PublishPayerEnvelopes)
NotificationApi — Full envelope stream for push notification servers (SubscribeAllEnvelopes)
GatewayApi — Client-to-gateway requests, replacing PayerApi (PublishClientEnvelopes, GetNodes). Reuses message types from payer_api.

ReplicationApi changes

Added SubscribeOriginators with dedicated request/response types for node-to-node subscription
Removed SubscribeAllEnvelopes (moved to NotificationApi, safe since unused)
Deprecated all other RPCs with comments pointing to their new service locations

Other changes

Deprecated PayerApi service
Removed google/api/annotations imports and HTTP option blocks from all xmtpv4 protos
Updated buf.yaml lint ignores for shared request/response types during transition

Verification

buf build passes
buf breaking passes (no breaking changes detected)
No google/api/annotations remain in xmtpv4

Closes #327 (Phase 1)

Note

Restructure d14n service protos into dedicated API services

Splits ReplicationApi into focused services: QueryApi, PublishApi, NotificationApi, and a new GatewayApi for client-facing operations.
Adds SubscribeOriginators RPC to ReplicationApi with a new SubscribeOriginatorsRequest supporting per-originator node filters and a cursor; marks several existing RPCs as deprecated.
Removes HTTP/REST google.api.annotations bindings from MisbehaviorApi, MetadataApi, and PayerApi, leaving only gRPC method signatures.
Marks PayerApi itself as deprecated; its PublishClientEnvelopes and GetNodes RPCs are now exposed via the new GatewayApi.
Behavioral Change: SubscribeAllEnvelopes is removed from ReplicationApi and moved to the new NotificationApi service.

^{Macroscope summarized fcad8b1.}

…onApi

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

neekolas · 2026-04-01T20:57:47Z

Phase 1: d14n service restructuring #328 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Wrap request fields in OriginatorFilter inner message and response envelopes in a single-item oneof for consistency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

macroscopeapp · 2026-04-02T03:53:06Z

Approvability

Verdict: Needs human review

This is a significant API restructuring that splits ReplicationApi into multiple focused services (QueryApi, PublishApi, NotificationApi, GatewayApi) and deprecates existing endpoints. While the author owns all files and changes are to proto schemas (not runtime code), the architectural scope of this 'Phase 1' restructuring warrants human review to ensure alignment with the intended service decomposition strategy.

^{You can customize Macroscope's approvability policy. Learn more.}

github-actions · 2026-04-06T22:24:48Z

🎉 This PR is included in version 3.89.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

…ors (#1917) ## Summary Phase 1 of the d14n service restructuring ([xmtp/proto#327](xmtp/proto#327)). Registers the new gRPC services introduced by [xmtp/proto#328](xmtp/proto#328) on the xmtpd server and implements the only new RPC method, `SubscribeOriginators`. ### New services registered | Service | Server | Backed by | Methods | |---------|--------|-----------|---------| | `QueryApi` | Replication | `message.Service` | `QueryEnvelopes`, `SubscribeTopics`, `GetInboxIds`, `GetNewestEnvelope` | | `PublishApi` | Replication | `message.Service` | `PublishPayerEnvelopes` | | `NotificationApi` | Replication | `message.Service` | `SubscribeAllEnvelopes` | | `GatewayApi` | Gateway | `payer.Service` | `PublishClientEnvelopes`, `GetNodes` | No new structs — the existing service structs satisfy the new interfaces because the proto changes use identical method signatures and request/response types. ### New RPC: `SubscribeOriginators` Added to `ReplicationApi`. Subscribes to envelopes from specific originator nodes with cursor-based catch-up, using the same `subscribeWorker` infrastructure as `SubscribeEnvelopes`. - `LastSeen` cursor is **required** (nil returns `InvalidArgument`) - Responses use the `Envelopes` oneof variant, leaving room for future status signals - Envelope batching accounts for the oneof wrapper overhead to avoid exceeding gRPC payload limits ### Internal type: `subscribeFilter` Replaced internal usage of the deprecated `message_api.EnvelopesQuery` proto with `subscribeFilter`, an unexported type with explicit `catchUpMode` enum: ```go type catchUpMode int const ( catchUpNone catchUpMode = iota // stream new envelopes only catchUpFromStart // catch up from the very beginning catchUpFromCursor // catch up from specified cursor position ) ``` Each subscribe endpoint sets the mode explicitly at its entry point — no implicit nil cursor semantics. ### Shared subscribe helpers Extracted `batchAndSendEnvelopes` and `catchUpWithSendFn` into `subscribe_common.go`, eliminating ~200 lines of duplication between `SubscribeEnvelopes` and `SubscribeOriginators`. Both use callback-based response wrapping so the batching, cursor dedup, and pagination logic is written once. ### Deprecated endpoints All deprecated `ReplicationApi` and `PayerApi` endpoints remain fully functional. SA1019 lint warnings for deprecated type usage are suppressed with a scoped exclusion in `.golangci.yaml` — these will be cleaned up in Phase 2 when deprecated endpoints are removed. ## Test plan - [x] `TestSubscribeOriginators` — filters by originator, catches up from empty cursor, streams live - [x] `TestSubscribeOriginators_NilFilter` — returns `InvalidArgument` - [x] `TestSubscribeOriginators_EmptyOriginatorIds` — returns `InvalidArgument` - [x] `TestSubscribeOriginators_NilLastSeen` — returns `InvalidArgument` (cursor required) - [x] `TestSubscribeOriginators_FromCursor` — catches up past cursor, then streams live - [x] `TestQueryApi_QueryEnvelopes` — new service endpoint returns envelopes - [x] `TestQueryApi_GetInboxIds` — new service endpoint is reachable - [x] `TestPublishApi_PublishPayerEnvelopes` — new service endpoint routes correctly - [x] `TestGatewayApi_GetNodes` — returns same response as deprecated `PayerApi.GetNodes` - [x] All 94 existing + new tests pass - [x] `go build ./...` clean - [x] `dev/lint-fix` clean (0 issues) 🤖 Generated with [Claude Code](https://claude.com/claude-code)    > [!NOTE] > ### Register new d14n gRPC services and implement `SubscribeOriginators` endpoint > - Adds `SubscribeOriginators` to `ReplicationApi` ([subscribe_originators.go](https://github.com/xmtp/xmtpd/pull/1917/files#diff-a77ed6e8a3a598507d3073a7941663293d866e4ca492147442e3deb18f37e4bd)): streams envelopes by originator node ID with catch-up from cursor, periodic keepalives, and oversized-envelope skipping. > - Registers `QueryApi`, `PublishApi`, and `NotificationApi` handlers in both the node API server ([server.go](https://github.com/xmtp/xmtpd/pull/1917/files#diff-104a11712684e520f0affa95e6a053faeb713306a10f69ee99b9729cc9dd1996)) and test infrastructure ([api.go](https://github.com/xmtp/xmtpd/pull/1917/files#diff-9c8d5b82b5a6dd5f551c0a19ec4a8491c0318e4efe142779bb3edad24d9b476c)). > - Registers `GatewayApi` alongside `PayerApi` in the gateway builder ([builder.go](https://github.com/xmtp/xmtpd/pull/1917/files#diff-27a239accb0576c3da95c1fea41bd81227bc4355f38be11a66d7b15135ea8e61)); `payer.Service` now also satisfies `GatewayApiHandler`. > - Refactors internal subscribe/query methods to use a `subscribeFilter` struct instead of proto `EnvelopesQuery`, and extracts shared helpers `batchAndSendEnvelopes` and `catchUpWithSendFn` ([subscribe_common.go](https://github.com/xmtp/xmtpd/pull/1917/files#diff-c9ba1bfc58febac0c4fd7dc8805ba3b94783de3dfd139c36c5526705416d9c25)). > - Updates generated proto, gRPC, Connect-Go, and OpenAPI files to reflect the new service split (QueryApi, PublishApi, NotificationApi, GatewayApi) and the replacement of `SubscribeAllEnvelopes` on `ReplicationApi` with `SubscribeOriginators`. > - Behavioral Change: `SubscribeAllEnvelopes` moves to `NotificationApi`; callers using `ReplicationApi.SubscribeAllEnvelopes` must migrate to `NotificationApiClient`. > >  > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 6ae509c.</sup> >  >   --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

neekolas and others added 7 commits April 1, 2026 13:49

feat: add SubscribeOriginators, deprecate relocated RPCs on Replicati…

6d1aa8d

…onApi

feat: create PublishApi and NotificationApi services

1fb92b0

feat: create QueryApi service for client-to-node queries

40f9cd0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat: create GatewayApi, deprecate PayerApi

c36ea8f

chore: update buf.yaml lint ignores for new service files

6f81eff

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: remove google/api/annotations from all xmtpv4 protos

167dd1d

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Gitignore plans folder

c270421

neekolas changed the title ~~feat: add SubscribeOriginators, deprecate relocated RPCs on ReplicationApi~~ Phase 1: d14n service restructuring Apr 1, 2026

neekolas mentioned this pull request Apr 1, 2026

Phase 2: d14n service restructuring — remove deprecated services and RPCs #329

Open

5 tasks

refactor: align SubscribeOriginators types with SubscribeTopics pattern

fcad8b1

Wrap request fields in OriginatorFilter inner message and response envelopes in a single-item oneof for consistency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

neekolas mentioned this pull request Apr 2, 2026

feat: register new d14n gRPC services and implement SubscribeOriginators xmtp/xmtpd#1917

Merged

12 tasks

neekolas marked this pull request as ready for review April 2, 2026 03:52

neekolas requested a review from a team as a code owner April 2, 2026 03:52

fbac approved these changes Apr 2, 2026

View reviewed changes

neekolas merged commit e3df70c into main Apr 3, 2026
9 checks passed

neekolas deleted the 04-01-phase_1_of_overhaul_of_d14n branch April 3, 2026 06:53

github-actions Bot added the released label Apr 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Phase 1: d14n service restructuring#328

Phase 1: d14n service restructuring#328
neekolas merged 8 commits into
mainfrom
04-01-phase_1_of_overhaul_of_d14n

neekolas commented Apr 1, 2026 •

edited by macroscopeapp Bot

Loading

Uh oh!

neekolas commented Apr 1, 2026

Uh oh!

macroscopeapp Bot commented Apr 2, 2026

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

neekolas commented Apr 1, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New services

ReplicationApi changes

Other changes

Verification

Restructure d14n service protos into dedicated API services

Uh oh!

neekolas commented Apr 1, 2026

Uh oh!

macroscopeapp Bot commented Apr 2, 2026

Approvability

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

neekolas commented Apr 1, 2026 •

edited by macroscopeapp Bot

Loading