d14n service renames

[neekolas]

Summary

We've learned a lot about the API surface of xmtpv4 since it was initially designed. I think we can structure the services to have clearer separations between different consumers.

Specific Problems

Multiple consumers of a single service

Currently xmtp.xmtpv4.message_api.ReplicationApi covers multiple consumers:

• Client -> Node
• Gateway -> Node (PublishPayerEnvelopes)
• Node -> Node (SubscribeEnvelopes)

We have different expectations for these consumers. Node -> Node requests expect an authentication token and no rate limiting. Gateway -> Node requests don't currently have authentication (although I think they could) and no rate limiting. Client requests are unauthenticated but rate limited.

Multiple uses of a single API

SubscribeEnvelopes is used in very different ways. Need to fully deprecate and break up into SubscribeTopics and SubscribeOriginators

Unclear naming

• We renamed the Payer API to Gateway API everywhere else but never updated the protos
• The Replication API covers a lot more than just replication

Proposed Solution

We could fix all these problems without breaking wire format compatibility in the short term, and then do a breaking change in a month when everyone is using a newer client version.

New Service Structure

message_api would contain multiple GRPC services

• ReplicationApi Only used for Node <> Node. Requires Node auth token
• QueryApi For Client -> Node (all queries and subscriptions, no auth token required)
• PublishApi For Gateway -> Node (PublishPayerEnvelopes)
• NotificationApi For push notification servers subscribing to all messages

gateway_api
Single service that is the same as payer_api today. Named gateway_api.GatewayApi. Used for client -> gateway requests

metadata_api and misbehavior_api are unchanged.

Migration Plan

Phase 1

1. Add SubscribeOriginators to the ReplicationApi. Make SubscribeEnvelopes internally proxy to SubscribeTopics or SubscribeOriginators based on query
2. Create QueryApi and PublishApi.
3. Mark all other endpoints on ReplicationApi as deprecated but leave in place. Internally proxy to QueryApi and PublishApi
4. Duplicate payer_api as gateway_api. Mark the old one as deprecated
5. Update server to handle all endpoints
6. Make client stop using deprecated endpoints and start using the new ones
7. Put the brand new and unused SubscribeAllEnvelopes into a new NotificationApi service
8. Start applying rate limits to all of QueryApi

Phase 2

1. Remove all deprecated endpoints from the server
2. Delete payer_api
3. Enforce that all requests to ReplicationApi have an auth token

Existing API

xmtp.xmtpv4.message_api.ReplicationApi

RPC	Request	Response	URL Path
SubscribeEnvelopes	SubscribeEnvelopesRequest	stream SubscribeEnvelopesResponse	/xmtp.xmtpv4.message_api.ReplicationApi/SubscribeEnvelopes
SubscribeAllEnvelopes	SubscribeAllEnvelopesRequest	stream SubscribeEnvelopesResponse	/xmtp.xmtpv4.message_api.ReplicationApi/SubscribeAllEnvelopes
SubscribeTopics	SubscribeTopicsRequest	stream SubscribeTopicsResponse	/xmtp.xmtpv4.message_api.ReplicationApi/SubscribeTopics
QueryEnvelopes	QueryEnvelopesRequest	QueryEnvelopesResponse	/xmtp.xmtpv4.message_api.ReplicationApi/QueryEnvelopes
PublishPayerEnvelopes	PublishPayerEnvelopesRequest	PublishPayerEnvelopesResponse	/xmtp.xmtpv4.message_api.ReplicationApi/PublishPayerEnvelopes
GetInboxIds	GetInboxIdsRequest	GetInboxIdsResponse	/xmtp.xmtpv4.message_api.ReplicationApi/GetInboxIds
GetNewestEnvelope	GetNewestEnvelopeRequest	GetNewestEnvelopeResponse	/xmtp.xmtpv4.message_api.ReplicationApi/GetNewestEnvelope

xmtp.xmtpv4.message_api.MisbehaviorApi

RPC	Request	Response	URL Path
SubmitMisbehaviorReport	SubmitMisbehaviorReportRequest	SubmitMisbehaviorReportResponse	/xmtp.xmtpv4.message_api.MisbehaviorApi/SubmitMisbehaviorReport
QueryMisbehaviorReports	QueryMisbehaviorReportsRequest	QueryMisbehaviorReportsResponse	/xmtp.xmtpv4.message_api.MisbehaviorApi/QueryMisbehaviorReports

xmtp.xmtpv4.metadata_api.MetadataApi

RPC	Request	Response	URL Path
GetSyncCursor	GetSyncCursorRequest	GetSyncCursorResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetSyncCursor
SubscribeSyncCursor	GetSyncCursorRequest	stream GetSyncCursorResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/SubscribeSyncCursor
GetVersion	GetVersionRequest	GetVersionResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetVersion
GetPayerInfo	GetPayerInfoRequest	GetPayerInfoResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetPayerInfo

xmtp.xmtpv4.payer_api.PayerApi

RPC	Request	Response	URL Path
PublishClientEnvelopes	PublishClientEnvelopesRequest	PublishClientEnvelopesResponse	/xmtp.xmtpv4.payer_api.PayerApi/PublishClientEnvelopes
GetNodes	GetNodesRequest	GetNodesResponse	/xmtp.xmtpv4.payer_api.PayerApi/GetNodes

Final Structure

xmtp.xmtpv4.message_api.ReplicationApi

Node <> Node. Requires node auth token.

RPC	Request	Response	URL Path
SubscribeOriginators	SubscribeOriginatorsRequest	stream SubscribeOriginatorsResponse	/xmtp.xmtpv4.message_api.ReplicationApi/SubscribeOriginators

xmtp.xmtpv4.message_api.NotificationApi

Full envelope stream.

RPC	Request	Response	URL Path
SubscribeAllEnvelopes	SubscribeAllEnvelopesRequest	stream SubscribeEnvelopesResponse	/xmtp.xmtpv4.message_api.NotificationApi/SubscribeAllEnvelopes

xmtp.xmtpv4.message_api.QueryApi

Client -> Node. No auth token required.

RPC	Request	Response	URL Path
QueryEnvelopes	QueryEnvelopesRequest	QueryEnvelopesResponse	/xmtp.xmtpv4.message_api.QueryApi/QueryEnvelopes
SubscribeTopics	SubscribeTopicsRequest	stream SubscribeTopicsResponse	/xmtp.xmtpv4.message_api.QueryApi/SubscribeTopics
GetInboxIds	GetInboxIdsRequest	GetInboxIdsResponse	/xmtp.xmtpv4.message_api.QueryApi/GetInboxIds
GetNewestEnvelope	GetNewestEnvelopeRequest	GetNewestEnvelopeResponse	/xmtp.xmtpv4.message_api.QueryApi/GetNewestEnvelope

xmtp.xmtpv4.message_api.PublishApi

Gateway -> Node.

RPC	Request	Response	URL Path
PublishPayerEnvelopes	PublishPayerEnvelopesRequest	PublishPayerEnvelopesResponse	/xmtp.xmtpv4.message_api.PublishApi/PublishPayerEnvelopes

xmtp.xmtpv4.message_api.MisbehaviorApi

Unchanged.

RPC	Request	Response	URL Path
SubmitMisbehaviorReport	SubmitMisbehaviorReportRequest	SubmitMisbehaviorReportResponse	/xmtp.xmtpv4.message_api.MisbehaviorApi/SubmitMisbehaviorReport
QueryMisbehaviorReports	QueryMisbehaviorReportsRequest	QueryMisbehaviorReportsResponse	/xmtp.xmtpv4.message_api.MisbehaviorApi/QueryMisbehaviorReports

xmtp.xmtpv4.metadata_api.MetadataApi

Unchanged.

RPC	Request	Response	URL Path
GetSyncCursor	GetSyncCursorRequest	GetSyncCursorResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetSyncCursor
SubscribeSyncCursor	GetSyncCursorRequest	stream GetSyncCursorResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/SubscribeSyncCursor
GetVersion	GetVersionRequest	GetVersionResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetVersion
GetPayerInfo	GetPayerInfoRequest	GetPayerInfoResponse	/xmtp.xmtpv4.metadata_api.MetadataApi/GetPayerInfo

xmtp.xmtpv4.gateway_api.GatewayApi

Client -> Gateway. Replaces payer_api.PayerApi.

RPC	Request	Response	URL Path
PublishClientEnvelopes	PublishClientEnvelopesRequest	PublishClientEnvelopesResponse	/xmtp.xmtpv4.gateway_api.GatewayApi/PublishClientEnvelopes
GetNodes	GetNodesRequest	GetNodesResponse	/xmtp.xmtpv4.gateway_api.GatewayApi/GetNodes

[github-actions]

🎉 This issue has been resolved in version 3.89.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀