Skip to content

tronscan: fix data loss in transaction/transfer pagination#1086

Open
KyJIJIEP wants to merge 4 commits into
zenmoney:masterfrom
KyJIJIEP:fix/tronscan-pagination-limits
Open

tronscan: fix data loss in transaction/transfer pagination#1086
KyJIJIEP wants to merge 4 commits into
zenmoney:masterfrom
KyJIJIEP:fix/tronscan-pagination-limits

Conversation

@KyJIJIEP

@KyJIJIEP KyJIJIEP commented Jun 29, 2026

Copy link
Copy Markdown

Summary

This PR fixes three pagination issues in the tronscan plugin that can cause operations to be silently dropped during sync, and aligns the plugin with the current limits of apilist.tronscanapi.com.

All changes are confined to src/plugins/tronscan/api.ts. Behaviour, converters and the public method signatures are unchanged.

Background

Tronscan's public API (apilist.tronscanapi.com) enforces limits that the plugin did not account for:

  1. List endpoints enforce start + limit <= 10000 per (address, time-range) query (max 50 records per page); paging past that ceiling is silently capped. This applies to transaction, token_trc20/transfers and transfer/trx alike — see the official docs.
  2. The account/tokens endpoint caps limit at 200 (default 20), per the official docs.

In addition, transfer/transaction list endpoints may return fewer rows per page than the requested limit (e.g. the long-standing behaviour of token_trc20/transfers, see tronscan/tronscan-api#59).

Problems fixed

  1. account/tokens requested limit: 999 — above the documented maximum of 200, so the request may be rejected or truncated. Lowered to 200.

  2. Records beyond the start + limit <= 10000 ceiling were lost. The previous loops paged by start and stopped when start + limit >= total. For an active wallet whose history within the requested range exceeds that ceiling, everything past it was never fetched. Now we page within a time window and, once a window reaches the ceiling, move its upper bound (end_timestamp) back to the oldest record we have seen and continue — so the full history is retrieved without gaps. A window stops as soon as it returns fewer records than the cap, so wallets below the ceiling cost exactly one pass.

  3. Rows between pages could be skipped. The cursor advanced by the requested limit (50). When Tronscan returned fewer rows than requested for a page, the next start jumped past the un-returned rows. The cursor now advances by the number of records actually returned.

Implementation

The three list methods (fetchTransactions, fetchTokenTransfers, fetchTronTransfers) now share a single typed fetchPaginated<Body, Item> helper that:

  • pages within a time window, advancing start by the actual returned count;
  • de-duplicates by record id (Set), so the boundary record re-fetched when the window narrows is not counted twice;
  • narrows the window via end_timestamp only when a window hits the start + limit <= 10000 ceiling, and guards against making no progress (records sharing a timestamp).

Testing

  • ts-standard src/plugins/tronscan/api.ts — passes (0 issues)
  • tsc --noEmit — passes (0 errors)

There are no existing unit tests for this plugin (no converters change here), so none were added; the change is isolated to the API/pagination flow per the repo's layering guidelines.

Align the Tronscan plugin with the current apilist.tronscanapi.com limits
and fix three pagination issues that could silently drop operations.

- account/tokens: lower limit from 999 to 500. Tronscan now caps this
  endpoint's limit at 500; oversized requests may be rejected/truncated.
- transaction & token_trc20/transfers: a single (address, time-range)
  query only returns the latest ~2000 records. The previous loop paged by
  `start` and trusted `total`, so anything beyond the cap was lost on busy
  wallets. Page within a time window, then move the window's upper bound
  (end_timestamp) back to the oldest record seen to reach older history.
- All list endpoints: advance the page cursor by the number of records
  actually returned instead of by the requested `limit`. Tronscan may serve
  fewer rows per page than requested (e.g. token_trc20/transfers), which
  previously caused rows between pages to be skipped.

Pagination logic is centralized in a single typed `fetchPaginated` helper
with id-based de-duplication for the window boundary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread src/plugins/tronscan/api.ts Outdated
The account/tokens endpoint caps `limit` at 200 (not 500), per the
official docs: docs.tronscan.org/en/api/account/account-tokens
(limit: integer, default 20, "Page size, max 200").
Comment thread src/plugins/tronscan/api.ts Outdated
The window paginator used a conservative cap of 2000, with a comment that
incorrectly claimed Tronscan returns "at most ~2000 records" per query.
Per the official docs, /api/transaction, token_trc20/transfers and
transfer/trx all enforce `start + limit <= 10000` (max 50 per page), with
no separate reduction for address-filtered queries. Raise the cap to 10000
to match the real ceiling: this does not fetch more data (a window stops as
soon as it returns fewer records than the cap), it only slides the time
window exactly at the API boundary instead of prematurely.
Comment thread src/plugins/tronscan/api.ts Outdated
Verified against the live API: list endpoints serve a full page (exactly
`limit`, max 50) until the data runs out, then a short page — the old
"returns fewer than requested" behaviour (tronscan/tronscan-api#59) no
longer reproduces, and reachable records are capped at `start + limit <=
10000` per query.

Drop the `windowCount` heuristic accordingly: the inner loop now stops on a
non-full page (window drained) and detects the ceiling via
`start + limit > 10000`, only then sliding the time window (end_timestamp =
oldest record seen) to collect entities beyond 10000. De-duplication is
kept because the offset is not perfectly stable across pages/boundaries.
The `extract` callback is simplified to return the item array directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants