tronscan: fix data loss in transaction/transfer pagination#1086
Open
KyJIJIEP wants to merge 4 commits into
Open
tronscan: fix data loss in transaction/transfer pagination#1086KyJIJIEP wants to merge 4 commits into
KyJIJIEP wants to merge 4 commits into
Conversation
Align the Tronscan plugin with the current apilist.tronscanapi.com limits and fix three pagination issues that could silently drop operations. - account/tokens: lower limit from 999 to 500. Tronscan now caps this endpoint's limit at 500; oversized requests may be rejected/truncated. - transaction & token_trc20/transfers: a single (address, time-range) query only returns the latest ~2000 records. The previous loop paged by `start` and trusted `total`, so anything beyond the cap was lost on busy wallets. Page within a time window, then move the window's upper bound (end_timestamp) back to the oldest record seen to reach older history. - All list endpoints: advance the page cursor by the number of records actually returned instead of by the requested `limit`. Tronscan may serve fewer rows per page than requested (e.g. token_trc20/transfers), which previously caused rows between pages to be skipped. Pagination logic is centralized in a single typed `fetchPaginated` helper with id-based de-duplication for the window boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhiriki
reviewed
Jun 30, 2026
The account/tokens endpoint caps `limit` at 200 (not 500), per the official docs: docs.tronscan.org/en/api/account/account-tokens (limit: integer, default 20, "Page size, max 200").
dzhiriki
reviewed
Jun 30, 2026
The window paginator used a conservative cap of 2000, with a comment that incorrectly claimed Tronscan returns "at most ~2000 records" per query. Per the official docs, /api/transaction, token_trc20/transfers and transfer/trx all enforce `start + limit <= 10000` (max 50 per page), with no separate reduction for address-filtered queries. Raise the cap to 10000 to match the real ceiling: this does not fetch more data (a window stops as soon as it returns fewer records than the cap), it only slides the time window exactly at the API boundary instead of prematurely.
dzhiriki
reviewed
Jun 30, 2026
Verified against the live API: list endpoints serve a full page (exactly `limit`, max 50) until the data runs out, then a short page — the old "returns fewer than requested" behaviour (tronscan/tronscan-api#59) no longer reproduces, and reachable records are capped at `start + limit <= 10000` per query. Drop the `windowCount` heuristic accordingly: the inner loop now stops on a non-full page (window drained) and detects the ceiling via `start + limit > 10000`, only then sliding the time window (end_timestamp = oldest record seen) to collect entities beyond 10000. De-duplication is kept because the offset is not perfectly stable across pages/boundaries. The `extract` callback is simplified to return the item array directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes three pagination issues in the tronscan plugin that can cause operations to be silently dropped during sync, and aligns the plugin with the current limits of
apilist.tronscanapi.com.All changes are confined to
src/plugins/tronscan/api.ts. Behaviour, converters and the public method signatures are unchanged.Background
Tronscan's public API (
apilist.tronscanapi.com) enforces limits that the plugin did not account for:start + limit <= 10000per(address, time-range)query (max 50 records per page); paging past that ceiling is silently capped. This applies totransaction,token_trc20/transfersandtransfer/trxalike — see the official docs.account/tokensendpoint capslimitat 200 (default 20), per the official docs.In addition, transfer/transaction list endpoints may return fewer rows per page than the requested
limit(e.g. the long-standing behaviour oftoken_trc20/transfers, see tronscan/tronscan-api#59).Problems fixed
account/tokensrequestedlimit: 999— above the documented maximum of 200, so the request may be rejected or truncated. Lowered to200.Records beyond the
start + limit <= 10000ceiling were lost. The previous loops paged bystartand stopped whenstart + limit >= total. For an active wallet whose history within the requested range exceeds that ceiling, everything past it was never fetched. Now we page within a time window and, once a window reaches the ceiling, move its upper bound (end_timestamp) back to the oldest record we have seen and continue — so the full history is retrieved without gaps. A window stops as soon as it returns fewer records than the cap, so wallets below the ceiling cost exactly one pass.Rows between pages could be skipped. The cursor advanced by the requested
limit(50). When Tronscan returned fewer rows than requested for a page, the nextstartjumped past the un-returned rows. The cursor now advances by the number of records actually returned.Implementation
The three list methods (
fetchTransactions,fetchTokenTransfers,fetchTronTransfers) now share a single typedfetchPaginated<Body, Item>helper that:startby the actual returned count;Set), so the boundary record re-fetched when the window narrows is not counted twice;end_timestamponly when a window hits thestart + limit <= 10000ceiling, and guards against making no progress (records sharing a timestamp).Testing
ts-standard src/plugins/tronscan/api.ts— passes (0 issues)tsc --noEmit— passes (0 errors)There are no existing unit tests for this plugin (no
converterschange here), so none were added; the change is isolated to the API/pagination flow per the repo's layering guidelines.