If we are to make the SDKs fork testing a better experience, it needs to be a fluid experience. The way we have addressed the need for historical state today in the Soroban Rust SDK and stellar-cli's snapshot testing is for the stellar-cli to download a history archive, which takes considerable effort to identify ahead of time the appropriate footprint. It also requires significant bandwidth.

This issue is to explore improving the fork testing experience by having the SDK act independently to retrieve ledger entries from data sources as it needs them, build a local footprint snapshot that can still be committed and used for subsequent runs, but where the footprint does not need to be known ahead of the first run.

The solution should have some properties:

• Reduced bandwidth for the common cases (the solution today requires a full archive to be downloaded).
• Not require the user to navigate footprints on their own (the solution today requires a user to identify a superset of their footprint).

The way the SDK fork testing works today is:

sequenceDiagram
    user->>+cli: create snapshot with<br/>accountIDs and contract IDs
    cli->>+historyarchive: req full ledger
    historyarchive->>-cli: full ledger (GBs, slow)
    cli->>cli: filter ledger down to<br/>accountIDs and contract IDs<br/>(painful dev experience)
    cli->>-user: snapshot (KBs)
    
    user->>+sdk: write test importing snapshot for fork testing
    sdk->>sdk: loads snapshot and uses it as storage starting point
    sdk->>-user: 

At a high level the way it should work:

sequenceDiagram
    user->>+sdk: write test fork testing ledger N
    loop 
    sdk->>sdk: test running
    alt local snapshot exists (cache)
    sdk->>sdk: load ledger entry for key K from snapshot
    else local snapshot does not exist
    sdk->>+datasource: req ledger entry for key K ledger N
    datasource->>-sdk: entry for K (Bytes)
    end
    end
    opt if new entries were downloaded
    sdk->>sdk: write local snapshot for starting point
    end
    sdk->>-user: 

This idea was first discussed here:

• Support forking from mainnet (or any target network) quickstart#625 (comment)

This issue should orient around two user stories:

1. Debugging a specific transaction – A developer wishes to write tests or debug behaviour before or after a specific transaction. Importantly this means state needs to be accessibly granular down to the level of a tx itself, not a ledger.

2. Testing against any arbitrary recent state – A developer wishes to write a test with mainnet data, but no specific point is particularly important.

The solutions for these two use cases could be different.