Proposal for getEvents v2 endpoint #1872
Replies: 18 comments 68 replies
|
How do you express a topic filter that is "all TRANSFER events to 0xme"? |
|
I had AI do a deep dive into all of the GitHub issues and even our own Discord to find all of the complaints and issues that we've heard over the last few years and matched it against this proposal. The results are here in this Gist: https://gist.github.com/kalepail/f0b14c806f01b69817d128504d0ec523 The TLDR is that it seems like what's being proposed is a good solution. And personally, as I reviewed it, I'm pretty excited about it. Great work. |
|
Quick clarification: for order=desc, is the ordering applied to the full event position tuple (ledger, tx_index, op_index, event_index) (i.e. fully reversed) or the intent is to only reverse ledger order while keeping tx/event order ascending within each ledger? Asking because this may impact cursor and index implementation. |
Are there two ways to do cursor-like pagination? I see the boundary type supports an event ID, and separately there's a cursor. What are the planned use cases for the former? |
It looks like the proposal introduces a new lever to control whether queries should be inclusive or exclusive of inputs. I don't think I've ever seen this in an API before, it seems novel. What problem is it solving for? It's not clear to me which problem at the start of the proposal it aligns with. When the min/max is a ledger I think inclusive makes sense as a default. And if someone wants exclusive it is very easy to plus or minus one. The event ID, I'm not clear on what the use case is for that and started a separate thread about that: #1872 (comment). If the goal is pagination, those inputs should probably be always exclusive. Unless the goal is to get a single event, in which case it might be clearer to offer a |
This comment has been hidden.
This comment has been hidden.
|
The events system needs to be more robust. I agree with several points raised so far, but the most critical issue is the 7-day retention window. It needs to be removed. Being able to reliably fetch historical events is mandatory for any real application. A 7-day limit is a major blocker for app developers. Apps are not temporary experiments. They are meant to live for a long time (forever?), and require durable event access for indexing, recovery, analytics, and user state. In our case, we are building a privacy pool. By design, we do not want to rely on a backend, because users should not have to trust any backend service. The RPC must be the source of truth. Short-lived events or reliance on third-party infrastructure directly break this model. I understand and support the goal of decentralization and not forcing the Stellar Foundation to operate heavy indexers indefinitely. However, the ecosystem is (unfortunately) still small ! I asked on Discord and was pointed to the Providers page.
Needing to deploy a backend just to fetch historical events significantly slows development and increases complexity. For early-stage builders, this friction is often enough to push them to another chain. |
|
IIRC topics are |
|
I think it would be valuable to elaborate on the feature subset we plan to support for full history: it seems this v2 variant has both way more features yet also less than the original. If we could compare and contrast it with what's feasible for full history, it may clue us in better as to what can be pared down from this endpoint. |
|
This is a timely proposal from my perspective and I look forward to implementing it for use in my sparse history Pakana Node. |
|
Couple of random thoughts related to above:
|
|
Starting a dedicated thread for nesting, following up on @leighmcculloch's comment here. Arrays of arrays tend to be confusing and there's a lot of potential nesting in the current structure. The following proposal replaces the current Definitionstype Filter = { // AND between txhash, contract id and all topic values
txHash?: string;
contractId?: string;
type?: "contract" | "system";
topics?: ScVal[]; //no further nesting
};
type FiltersQuery = {
filters: Filter[]; // UNION between filters
};Example, XLM transfers to and from a specific address{
filters: [
{
contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
type: "contract",
topics: [{ symbol: "transfer"}, { address: "GABC..." }]
},
{
contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
type: "contract",
topics: [{ symbol: "transfer"}, "*", { address: "GABC..." }]
}
]
}Example with topicN instead of topics arrayI find the topic array and the wildcards somewhat confusing so prefer a topicN approach as follows. Though not religious about it. {
filters: [
{
contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
type: "contract",
topic0: { symbol: "transfer"},
topic1: { address: "GABC..."}
},
{
contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
type: "contract",
topic0: { symbol: "transfer"},
topic2: { address: "GABC..."}
}
]
} |
|
Below I describe a search-style query language for the events endpoint that I think we could consider and has advantages with regards to straddling the simple vs expressive boundary. The API is similar in experience to the GitHub Search API. Note There's an example of a query syntax API like the one described here at: One of the challenges with our existing API and the proposals we've been discussing is that we are constantly trading between two things:
In a fixed schema request format these two things are at odds, because the more powerful the filter parameters are, the more challenging it is for humans and machine to understand and due to the fixed nature of the request that complexity is always exposed. (e.g. the nested topics) Search-style query languages can be expressive while still being very easy to use, especially for basic queries. This is because they naturally provide gradual complexity to a user. For example, for Stellar events, this would mean replacing the Some keys that I expect would be supported:
All contract events from a specific ledger: Transfer events for a contract: Events from a specific transaction: Transfer, mint, clawback, and burn events for XLM and USDC: Events where a specific address appears in any topic position, and noting how Query syntaxes come with their own challenges. Without good documentation it is often hard for a developer to learn exactly how to query something because there is no inherent structure that captures the full scope of all possibilities. Good documentation is super important for supporting the advanced use cases. An API shaped like this can still have limits, expressed either as generically or specifically as necessary. For example:
|
|
Have we considered a minimal modification to getEvents? Taking a step back it seems to me like some of the problems could be solve minimally in ways that are almost backwards compatible, without significant changes to the API, and maybe that would be sufficient. For example, all the following changes make subtle changes, some technically breaking, but all in very forgiving ways, without a significantly new endpoint.
Assume
Is this actually a problem? Seems like an internal concern. Either way, make it opaque.
Error on endLedger passed if already paginating an earlier query.
The proposal with hasMore: true + scannedLedger showing scan progress sounds good, and we can add those fields to the pagination section of the existing response.
Add as new top level field without any other structural changes, and error 400 if combined in incompatible ways.
order: "desc" iterates newest-first.
This issue isn't listed in the top of the documents problem list, but the getEvents method requiring either
This issue isn't listed in the top of the documents problem list, but it came up in the discussion. The (I realise I'm posting a couple of suggestions 1 2 at either end of the spectrum.) |
|
Hey! Any news on this? 😊 |
|
I've rewritten the proposal at the top of this discussion, incorporating the feedback from the different threads: Filters now use the flat shape @tomerweller proposed: a One open question, do we need to filter by event type (e.g. Also, note that filters are limited by the number of distinct index terms used. ScVal encoding is explicit. Following @leighmcculloch's observation that some xdr-json values are bare strings, value encoding is no longer inferred from shape: Pagination is rebuilt. The The transaction mode is removed in favor of a companion change: getTransaction gains an The event object is unchanged from v1 except that the deprecated Please review the latest proposal and let me know what you think! |
|
Thinking about how the JS SDK will support v2. Since v2 changes the request shape (min/max, inclusive bounds) and unknown fields are rejected as invalid_params, v1 and v2 requests are mutually incompatible if they share a method name. During the adoption window, SDKs and multi-provider apps will hit a mix of upgraded and non-upgraded nodes, so we need a cheap way to detect support. 2 qs
|
|
The design and API for getEvents V2 has been finalized so I will close the discussion. The work to implement the endpoint is tracked here stellar/stellar-rpc#774 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Overview
This proposal specifies an improved
getEventsStellar RPC API for querying events on the Stellar blockchain. It is designed to work identically over a 7-day retention window and over full history.Design Goals
The API addresses specific problems reported in GitHub issues stellar/stellar-rpc#426 and stellar/stellar-rpc#575:
topic0..topic3fields match topic positionsendLedgerignored during paginationoldestLedger,latestLedger)scanStatus+scannedLedgershow scan progressevents.parsedon getTransaction (companion change, below)order: "desc"iterates newest-firstfilterslist unions flat filters in one requestRequest
The API accepts two mutually exclusive request modes:
cursor)cursorpresentType Definitions
Field Definitions
minLedgernumbermaxLedgernumberorder"asc" | "desc""asc"(oldest first)filtersFilter[]limitnumbercursorstringxdrFormat"base64" | "json""base64"xdrInputFormat"base64" | "json"ScValvalues in the request. Default:"base64"An event matches the request when it matches at least one filter, and matches a filter when every specified field matches. Omitted fields are unconstrained.
Ledger bounds are inclusive on both ends. (v1's
endLedgerwas exclusive: a v1 request ported unchanged now includes one more ledger than before.) Bounds may extend beyond what the node holds; the few ranges rejected outright are listed under Errors.Empty values are invalid: omit
filtersentirely to match all events, and every filter must specify at least one field. A filter'stypematches the event's type label; note that fee and refund events carry typecontract. Unknown request fields are rejected: an omitted field is a wildcard here, so silently ignoring a misspelled field would silently widen the results.Filters
Each filter is a flat object with positional semantics:
topic0matchestopic[0],topic1matchestopic[1], and so on.filterslist are ORTopic values are encoded per
xdrInputFormat. Examples in this document use the JSON form and therefore setxdrInputFormat: "json":The examples below follow the SEP-41 token convention, where a transfer event's topics are:
Query patterns:
[{ topic0: transfer }][{ topic0: transfer, topic1: me }][{ topic0: transfer, topic2: me }][{ topic0: transfer, topic1: me }, { topic0: transfer, topic2: me }][{ topic0: transfer }, { topic0: approve }][{ contractId: A, topic0: transfer }, { contractId: B, topic0: swap }]Limits
filterstopic0..topic3)limitThe term budget counts how many distinct values a query asks the index about. Each distinct contract id is one term, each distinct type is one term, and each distinct value at a given topic position is one term. The same value at the same position is counted once no matter how many filters repeat it, while the same value at two different positions counts twice. For example,
[{ topic0: transfer, topic1: me }, { topic0: transfer, topic2: me }]uses 3 terms:transferattopic0,meattopic1, andmeattopic2. A five-branch token filter (transfer from/to, mint to, burn from, clawback from) uses 7: one contract id, fourtopic0symbols, and the account attopic1and attopic2. Queries over the budget returninvalid_paramswithtermsUsedandtermBudgetin the error data; split them into parallel queries and merge by event id (to checkpoint a split query, persist its least-advanced sub-query'sscannedLedger).RangeQuery Rules
asc(default)minLedgermaxLedger(none: follows the chain tip)descmaxLedger(the node's latest ledger at query start),minLedger(2, the genesis ledger)Rationale:
minLedger: ensures portable queries across RPCs with different retention (a full-history node and a 7-day node would otherwise silently answer differently)Bounds:
maxLedgermaxLedgeris the API's one open edge: it follows the chain tip forevermaxLedger, given or defaulted, is fixed when the query startsminLedgerolder than the node'soldestLedgeris legal, with or withoutmaxLedger: the node serves the ledgers it holdsmaxLedgeromitted andminLedgerabove the tip. Such a scan would start at the tip and descend, butminLedgerlies above its starting point, so nothing could ever matchExamples:
Response
eventscursor"COMPLETE", where it is omitted: an absent cursor means the query is finished. It carries the whole query (bounds, order, filters) and does not expirescanStatus"HAS_MORE": more to scan on this node right now; request the next page (a page can be empty when the internal scan limit was hit first; see Sparse data)."WAITING_FOR_LEDGERS": the query needs ledgers this node does not have yet; retry the same cursor later."OLDEST_REACHED": the query wants history older than this node holds; you have everything this node can give, but the query is not complete."COMPLETE": the query is finished and will never return more; the response carries no cursorscannedLedgerscannedLedgernames the last fully delivered ledger and the cursor resumes inside the partial one. If no ledger of the range is complete yet, it is the ledger just before the range:minLedger - 1ascending,maxLedger + 1descendingoldestLedgerlatestLedgerA scan that finishes its range exactly at the newest or oldest ledger the node holds reports
COMPLETE, notWAITING_FOR_LEDGERSorOLDEST_REACHED.The universal client loop:
Sparse data
Every request runs under an internal scan limit. The server scans a bounded amount of ledgers per request so that response time stays predictable no matter how rare the matching events are; the bound is set by the node operator.
limitcaps how many events a page may carry, not how much the server reads to find them.A query for rare events can therefore return an empty page: the scan hit its limit before finding a match.
scannedLedgerstill advances, and the next request continues from there. How far one request advances varies widely: a selective query can cheaply rule out long stretches of ledgers and jump far ahead, while a broad query moves a few hundred ledgers at a time:This means: no matches yet, and everything through ledger 400000 has been scanned. Keep following the cursor.
Event Schema
The event object is unchanged from the current getEvents response, except that v1's deprecated
inSuccessfulContractCallfield is dropped:Exactly one of
topic/topicJsonand one ofvalue/valueJsonis present, perxdrFormat. Clients must ignore unknown response fields and tolerate unknowntypevalues, so fields and event kinds can be added later without breaking anyone.Ordering and ids (behavior the current API has but never documented):
order: "desc"returns the exact reverseErrors
Errors are standard JSON-RPC 2.0 error objects.
error.datacarries a machine-readablereasonplus the additional fields listed per reason below, so clients can react without parsing message text. Request validation runs before any scanning, and no error consumes or advances a cursor:reasonerror.datainvalid_paramsledger_out_of_rangemissingLedger(the ledger the query needs that this node does not have),oldestLedger,latestLedgerminLedgerbelowoldestLedger(pruned or not yet backfilled;missingLedgerisminLedger), an ascending continuation whose next ledger fell belowoldestLedger(pruning overtook the cursor;missingLedgeris that next ledger), or a descendingminLedgerabovelatestLedgerwithmaxLedgeromitted (the defaultedmaxLedgeris the tip;missingLedgerisminLedger). Descending scans never get this error: they stop withOLDEST_REACHEDinsteadmissingLedgerbelowoldestLedger: retrying helps only on a node that holds, or backfills, older history.missingLedgerabovelatestLedger: retrying succeeds once this node holds ledgerminLedgercursor_malformedoldestLedger,latestLedgerConditions producing
invalid_paramscursorwith other query paramsmin > maxorder: "asc"withoutminLedgerlimit < 1or> 1000filtersempty or over 256typein a filtertermsUsed,termBudgetin data)xdrFormat/xdrInputFormatCompanion Change: getTransaction
getEvents v2 has no transaction mode. Instead, getTransaction and getTransactions add one field to the
eventsobject they already return:The existing fields carry raw, unparsed XDR blobs grouped by kind, with no event ids and no single ordering.
parsedcarries the same fee/refund and contract events decoded into theEventobjects defined above, in execution order, with the same ids they have in the getEvents stream, so a client needs no XDR decoding and can correlate directly with getEvents. Nothing existing moves or changes shape, so the addition breaks no client, and a transaction detail page becomes one call: status, result, and usable events together.parsedexcludes diagnostic events, which have no position in the event stream and therefore no ids; diagnostics remain available through the existing raw field and through simulateTransaction.Appendix: Code Examples
Latest N events:
Historical range with pagination:
Live tracking:
Resume from database:
Wallet activity (sent + received), one request:
Transaction detail page (companion change):
All reactions