Add tx-granular ledger snapshot source using meta, RPC, and archives

[leighmcculloch]

What

Add new crates for fetching ledger entries from multiple data sources to enable more seamless and built-in fork testing in the sdk. Add soroban-ledger-snapshot-source-tx for transaction-level snapshot sources.

sequenceDiagram
    autonumber
    actor App as 
    participant SelTxMeta as Query<br/>Ledger<br/>Tx Meta
    participant LedgerMeta as Query<br/>Ledger – 1..N<br/>Tx Meta
    participant Archive as History<br/>Archive
    participant RPC as RPC

    App->>SelTxMeta: Lookup with key
    Note over App,SelTxMeta: If found → use it.<br/>If not → continue.
    
    App->>SelTxMeta: Look in prior txs in same ledger
    Note over App,SelTxMeta: If found → use it.<br/>If not → continue.

    opt Optionally use RPC
        App->>RPC: getLedgerEntries([key])
        Note over App,RPC: Use if (lastModified < queryLedger)<br/>AND (rpcLatestSeen >= queryLedger).<br/>Otherwise continue.
    end

    App->>LedgerMeta: Look in prior ledger meta
    Note over App,LedgerMeta: If found → use it.<br/>If checkpoint ledger not reached → continue to next ledger meta<br/>If checkpoint ledger is reached → continue to archive.

    App->>Archive: Download checkpoint from history archive and search
    Note over App,Archive: If found → use it.<br/>If not → does not exist.

Loading

Why

The current fork testing experience utilising the stellar-cli has low granularity only at the boundaries of ledgers, and requires downloading full history archives and manually identifying footprints ahead of time, which is difficult to do well and a poor developer experience. This change enables the SDK to lazily fetch ledger entries on-demand from the most efficient source available, caching results locally for subsequent runs. Developers will be able to fork test against any ledger and transaction without pre-identifying the footprint.

The change uses a Ledger Meta Storage (SEP-54), an RPC, and a History Archive to collect ledger entries.

The change caches results in three layers. All raw files downloaded are cached in the system cache directory and reused across tests, across workspaces. All ledger entries found are cached in the system cache directory. All ledger entries found for the current workspace are cached in the tests-snapshot-source directory intended to be committed so that CI runs reproducibly without needing to collect entries. Note that the format of that cache is not a ledger snapshot json file because for many tests running concurrently one file per ledger entry is easier to manage.

Close #1448

Try it out

Add the following dependencies to your Cargo.toml:

[dev-dependencies]
soroban-sdk = { version = "25.1.0", features = ["testutils"] }
soroban-ledger-snapshot-source-tx = { git = "https://github.com/stellar/rs-soroban-sdk", branch = "snapshot-source-tx" }
bytes-lit = "0.0.6"

Note: Requires soroban-sdk v23.4.0 or later.

Example

use bytes_lit::bytes;
use soroban_ledger_snapshot_source_tx::{Network, TxSnapshotSource};
use soroban_sdk::{token::TokenClient, Address, Env};

#[test]
fn test_fork_at_tx() {
    // The address of the native asset (test XLM) on testnet.
    const NATIVE_ADDRESS: &str = "CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC";
    
    // Create a snapshot source at ledger 8655, just BEFORE the specified tx.
    let source = TxSnapshotSource::new(
        Network::testnet(),
        8655,
        Some(bytes!(0x580bad1826d02b45634a3742049a23bf652fb2d4bc0814c83f2f58a7a9810ac9)),
    );
    
    // Setup the environment with the source.
    let env = Env::from_ledger_snapshot(source);
    let contract = Address::from_str(&env, NATIVE_ADDRESS);
    let client = TokenClient::new(&env, &contract);

    // Lookup the balance of the following address.
    let addr = Address::from_str(&env, "CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD2KM");
    let bal = client.balance(&addr);
    assert_eq!(bal, 47);  // Balance just before the tx executed
}

Observing State Changes

The TxSnapshotSource looks up state with transaction-level granularity when a transaction hash is provided allowing developers to debug a transaction by starting at the point just before the transaction. For example:

Before a specific tx: Pass the tx hash to get state just before that tx executed:

let source = TxSnapshotSource::new(
    Network::testnet(),
    8655,
    Some(bytes!(0x580bad1826d02b45634a3742049a23bf652fb2d4bc0814c83f2f58a7a9810ac9)),
);
// Balance will be 47 (before this tx added 10)

End of ledger (after all txs): Use None for the tx_hash to get state at the end of the ledger:

let source = TxSnapshotSource::new(Network::testnet(), 8655, None);
// Balance will be 57 (after all txs in ledger 8655)

Different ledgers/txs: Update the test to investigate the state of the balance at each of the following ledgers and transactions to see how the transactions affected the balance. Check out the links to see the operations the transactions performed and how they align with the changes in balances observed.

Ledger	TX HASH	Change	Bal
8511	51b21c588f3b397f957d3c14685debf96224c27c5a45b81b27eb5ee0a56517bf	+ 1	1
8613	52e9a67fdd78105ed84f6d073f50a464e14c1c587a5e4b45842e4b3386336751	+ 2	3
8622	a5dd4e98df837da3b1a7de52e9698c1a28ae15973f52651c765f1654a3033993	+ 3	6
8632	7a6143351fc48aed1b34d9d0b1690bfe11404cc10cc10f5d52ec06758d0f2b5b	+ 6	12
8633	d5f676598fd331867f728c4f1c28ea6045c2d31a80d48bfb873d8302816a6304	+ 7	19
8633	34e420d6cf03ec2ad1560545f6f98ae281bbb6ac9837d23dbc44d158b2942a65	+ 8	27
8655	52ec73f485fb132fb5777c6f2be6c0358e9587c64f4a9ef1f861b56923493400	+11	38
8655	ccede7115f962f3680dc7ff45b21e52e5ea02004f6c2e5b47cf35b853a91203a	+ 9	47
8655	580bad1826d02b45634a3742049a23bf652fb2d4bc0814c83f2f58a7a9810ac9	+10	57

Debugging with RUST_LOG

Enable logging to see which data sources are being queried and what entries are found:

RUST_LOG=soroban_ledger_fetch=debug cargo test -- --nocapture

Example output:

2025-12-19T14:56:01.070297Z  INFO soroban_ledger_fetch: fetch, key: {"contract_data":{"contract":"CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC","durability":"persistent","key":"ledger_key_contract_instance"}}
2025-12-19T14:56:01.070932Z DEBUG soroban_ledger_fetch: fetch from meta range, count: 16, first: 8655, last: 8640
2025-12-19T14:56:01.071069Z DEBUG soroban_ledger_fetch: fetch from meta, ledger: 8655
2025-12-19T14:56:01.990395Z DEBUG soroban_ledger_fetch: fetch from rpc, ledger: 8655
2025-12-19T14:56:02.900716Z DEBUG soroban_ledger_fetch: found from rpc, last_modified: 689, usable: true
2025-12-19T14:56:02.919258Z  INFO soroban_ledger_fetch: found, entry: {"data":{"contract_data":{"contract":"CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC","durability":"persistent","ext":"v0","key":"ledger_key_contract_instance","val":{"contract_instance":{"executable":"stellar_asset","storage":[{"key":{"symbol":"METADATA"},"val":{"map":[{"key":{"symbol":"decimal"},"val":{"u32":7}},{"key":{"symbol":"name"},"val":{"string":"native"}},{"key":{"symbol":"symbol"},"val":{"string":"native"}}]}},{"key":{"vec":[{"symbol":"AssetInfo"}]},"val":{"vec":[{"symbol":"Native"}]}}]}}}},"ext":"v0","last_modified_ledger_seq":689}
2025-12-19T14:56:02.941928Z  INFO soroban_ledger_fetch: fetch, key: {"contract_data":{"contract":"CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC","durability":"persistent","key":{"vec":[{"symbol":"Balance"},{"address":"CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD2KM"}]}}}
2025-12-19T14:56:02.942165Z DEBUG soroban_ledger_fetch: fetch from meta range, count: 16, first: 8655, last: 8640
2025-12-19T14:56:02.942205Z DEBUG soroban_ledger_fetch: fetch from meta, ledger: 8655
2025-12-19T14:56:02.950580Z  INFO soroban_ledger_fetch: found, entry: {"data":{"contract_data":{"contract":"CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC","durability":"persistent","ext":"v0","key":{"vec":[{"symbol":"Balance"},{"address":"CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD2KM"}]},"val":{"map":[{"key":{"symbol":"amount"},"val":{"i128":"47"}},{"key":{"symbol":"authorized"},"val":{"bool":true}},{"key":{"symbol":"clawback"},"val":{"bool":false}}]}}},"ext":"v0","last_modified_ledger_seq":8655}

TODO

• Integration tests using quickstart.
  • Dependent on: Add galexie meta lake quickstart#835
    • Add Filesystem datastore support go-stellar-sdk#5892
    • Add Filesystem datastore support stellar-galexie#53
    • Add galexie service for local ledger meta store quickstart#870
• Reorganise code/crates into an arrangement that'll make sense longer term.
• Test coverage.
• Rust docs.
• Investigate if it would work to use the ledger snapshot JSON file format.
• Support providing connection and configuration.
  • In code.
  • Via environment variables.
  • With defaults or no defaults?
• Experiment with optimising the first-pass lookup by including in the lookup all the ledger entries in the footprint of the transaction to reduce the chance of needing to do more than one lookup.
• Experiment with including a lookup step that uses the Hubble BigQuery as proposed in Improve the in SDK fork testing experience #1448 (comment).
• Experiment with building an index either for the raw files, or across all files of a specific snapshot, to speed up follow up look ups.
• Incorporate this test from @earrietadev.
  • Evaluate the 24 min runtime and what opportunities exist to optimise.
• Developer docs with example.
• Is it intuitive that the ledger seq when specified means the start of the ledger, not end?

Thanks

Thanks @orbitlens for sharing the idea of using transaction meta as a way to collect recent state. Thanks to all the people who provided feedback to me about how they use the existing stellar snapshot create functionality.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	reqwest@​0.12.28
	thiserror@​2.0.18
	cargo_metadata@​0.19.2
	directories@​6.0.0
	flate2@​1.1.9
	tracing@​0.1.44
	tracing-subscriber@​0.3.23
	zstd@​0.13.3
	bytes-lit@​0.0.5
	url@​2.5.8

View full report

[earrietadev]

is it possible to set it so the timestamp is also set after the snapshot has been loaded? Currently we need to manually set it after the env is created from the snapshot with something like e.ledger().set_timestamp(1766260837);

[willemneal]

This seems like an interesting way of handling a simulation locally. Or at least doing a read call locally.

[orbitlens]

Looks awesome. Thanks, Leigh! ❤️

[willemneal]

This probably should be the cache crate's responsibility. If it is abstracted more than this crate will never need to use a fs. This way we could eventually use Wasm to run this in the browser.