feat(backend): per-network ledger cursor, gap alerts, reindex reset - #250
Merged
jhayniffy merged 2 commits intoMar 29, 2026
Merged
Conversation
- Add ledger_cursors (network PK, last_processed_ledger, updated_at) and ledger_gap_alert_dedup for cooldown-based gap notifications - Indexer advances cursor inside same Prisma tx as raw_events + projections - Gap alert when latestLedger - lastProcessed > threshold; dedupe via cooldown - Admin POST /admin/reindex resets cursor (fromLedger-1), enqueues Bull job; ReindexWorkerService runs processUntilCaughtUp (disabled in test / DISABLE_REINDEX_WORKER) - STELLAR_NETWORK + INDEXER_GAP_ALERT_* env; ClaimsService reads cursor by network - Migration seeds testnet row from legacy indexer_state - Tests: cursor resume, tx-bound advance, gap dedupe, admin reindex - Fix idempotency middleware Redis import path + fail-open on lookup errors - Jest: include *.spec.ts in testMatch Made-with: Cursor
|
@favourawaku Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
closes #213
Adds a per-network ledger cursor so the Soroban indexer can resume after restarts or RPC outages without reprocessing from genesis or skipping ledgers. Cursor updates are atomic with event upserts. Gap detection alerts when the chain head pulls too far ahead of the cursor, with cooldown-based deduplication. Admin reindex resets the cursor and enqueues a BullMQ catch-up job.
Changes
Database
ledger_cursors:network(PK),last_processed_ledger,updated_at.ledger_gap_alert_dedup: last gap alert time per network (cooldown / storm prevention).testnetfrom legacyindexer_statewhere present.Indexer
STELLAR_NETWORK(defaulttestnet).last_processed_ledgerupdated inside the same Prisma transaction asraw_events+ projection writes.latestLedger - lastProcessed > INDEXER_GAP_ALERT_THRESHOLD_LEDGERS(default100); repeats only afterINDEXER_GAP_ALERT_COOLDOWN_MS(default 1h).ReindexWorkerService: processesreindexqueue →processUntilCaughtUp(off inNODE_ENV=testorDISABLE_REINDEX_WORKER=1).handleClaimFiledand vote payload handling.Admin API
POST /admin/reindex: optionalnetwork; setslast_processed_ledger = max(0, fromLedger - 1), enqueues job, audit includesnetwork+jobId.Claims
getLastLedgerusesledger_cursorsfor the configured network, withindexer_statefallback.Config
STELLAR_NETWORK,INDEXER_GAP_ALERT_THRESHOLD_LEDGERS,INDEXER_GAP_ALERT_COOLDOWN_MSinenv.validation.ts.Tests & tooling
jest.config.js: include**/*.spec.ts.Deploy notes
Run
npx prisma migrate deploy(or equivalent) before starting workers.Testing
npm testandnpm run buildinbackend/.