[RelayMiner] Enable backing up in-memory session-trees to disk

[Olshansk]

Objective

1. Ensure Session Rollovers do not reduce overall network performance.
2. Guarantee that the solution to ☝️ does not reduce RelayMiner rewards.
3. Based on the implementation in feat: Add in-memory SMT storage option for RelayMiner #1730, enable performant RelayMiners that can also back up session trees to disk.

Origin Document

• Discord thread with poktscan (@jorgecuesta, @RawthiL, etc…)
• First implementation of a workaround: feat: Add in-memory SMT storage option for RelayMiner #1730, which produced the following outcome:

Goals

• Maintain the session rollover fix by preventing RelayMiner IOPS usage from blocking relays.
• Provide an "in-memory" mode that minimizes state loss on restart.
• Ensure disk-backed session trees are performant and do not bottleneck the relay path.
• Validate that RelayMiner rewards remain unaffected after these changes.

Deliverables

• Design and implement a solution that satisfies all goals above.
  • Periodic writing?
  • Process preemption with different priorities?
  • Other?
• Ensure an E2E test for session rollovers is included in CI.
• Search for and remove all TODO(#1734) markers in the codebase.
• Provide documentation for operators on when to use in-memory vs. disk-backed mode.
• Optional: Add benchmarks to measure relay throughput during session rollovers (with and without disk backup).

Non-goals / Non-deliverables

• Exploring alternative mechanisms for reducing heavy IOPS usage in RelayMiner (can be a follow-on ticket).
• Researching or adopting other KV store engines beyond those currently supported.

---

Creator: @Olshansk
Co-Owners: @bryanchriswhite @red-0ne @jorgecuesta

[bryanchriswhite]

I spent today largely rethinking the approach here, also discussing the issue with @red-0ne.

Given the performance benefit of using the in-memory SMST KVStore for claiming/proving, it seems like we should prefer to ALWAYS use an in-memory store for nominal operation (i.e. immediate write per mined relay and read for claim/proof generation). In order to resolve the backup/restore issue, I propose wrapping the sessionTree's treeStore in a new type, BackupKVStore.

BackupKVStore holds a reference to a primaryStore and a backupStore. It ALWAYS writes to both, but ONLY reads from primaryStore.

The backupStore would be populated with another new type, ThrottledKVStore, which wraps another KVStore and a config. ThrottlingKVStore batches writes to its internal KVStore according to its config.
(EDIT: upon further consideration, there doesn't seem to be a need for ThrottledKVStore. Diagrams below have been updated.)

Currently, on relayminer startup, the existing relayerSessionsManager#sessionSMTStore (session metadata) KVStore is used to reconstruct the relayerSessionsManager#sessionTrees. This proposal would extend this behavior to include the reconstruction and re-population (via iteration) of missing in-memory KVStores such that the normal claim/proof cycle can resume, using the in-memory SMST KVStore.

---
title: Relayminer SMST Backup Flow
---

flowchart

subgraph relayerSessionsManager
    sessionTrees
end

subgraph BackupKVStore
    smtsimplemap.simpleMap
    %% ThrottledKVStore
    pebble.pebbleKVStore 
end

%% subgraph ThrottledKVStore
%%    pebble.pebbleKVStore 
%% end

sessionTrees -..-> |**write** on mined relay| smtsimplemap.simpleMap
sessionTrees -.-> |**write** on mined relay| pebble.pebbleKVStore:::newlink
sessionTrees --> |"(contains - per session)"| BackupKVStore:::newlink

relayerSessionsManager -.-> |**read** for claiming/proving| smtsimplemap.simpleMap

classDef newlink stroke:#2ecc71,stroke-width:2px,color:#2ecc71;
class BackupKVStore,ThrottledKVStore newlink;

linkStyle 1 color: #2ecc71

Loading

---
title: Relayminer Restore Flow
---

flowchart

subgraph relayerSessionsManager
    sessionTrees
end

subgraph sst["sessionSMTStore (sesssion metadata store)"]
      pb2[pebble.pebbleKVStore]
end

relayerSessionsManager ---> |"1. Load sessions metadata - loadSessionTreeMap()"| pb2
sst -.-> |"2. Restore sessionTrees **without SMST**"| sessionTrees

subgraph BackupKVStore
    subgraph bs["**primaryStore**"]
        sm[smtsimplemap.simpleMap]
    end
    
    subgraph ps["**backupStore**"]
        pb[pebble.pebbleKVStore]
    end
end

relayerSessionsManager --> |"3. Iterate to restore in-memory KVStore"| pb
pb -.-> |"(restore SMST KVStore)"| sm
sm -.-> |"4. Import SMSTs into their sessionTrees - importSessionTree()"| sessionTrees


classDef newlink stroke:#2ecc71,stroke-width:2px,color:#2ecc71;
class BackupKVStore,ps,pb,ThrottledKVStore newlink;

linkStyle 2,3 color: #2ecc71

Loading

classDiagram

class relayerSessionsManager {
    sessionTrees SessionsTreesMap
    sessionSMTStore pebble.PebbleKVStore
}

%% class SessionTree {
%%     <<interface>>
%%     Update(key, value []byte, weight uint64) error
%% }

relayerSessionsManager --* sessionTree
%% relayerSessionsManager --* SessionTree

class sessionTree {
    treeStore smtkvstore.MapStore
}

%% sessionTree --|> SessionTree

class smtkvstore.MapStore {
	<<interface>>
    Get(key []byte) ([]byte, error)
	Set(key, value []byte) error
	Delete(key []byte) error
	Len() (int, error)
	ClearAll() error
}

sessionTree --* smtkvstore.MapStore

class BackupKVStore:::newlink {
    primaryStore smtkvstore.MapStore
    backupStore smtkvstore.MapStore
}

BackupKVStore --|> smtkvstore.MapStore

%% class ThrottledKVStore:::newlink {
%%     config batchingConfig
%% }

%% class throttledKVStoreConfig:::newlink {
%%     maxAccValuesSizeBytes uint64
%%     maxBatchSize uint64
%%     maxWait time.Duration
%% }

%% ThrottledKVStore --* throttledKVStoreConfig
%% ThrottledKVStore --|> smtkvstore.MapStore

classDef newlink stroke:#2ecc71,stroke-width:2px,color:#2ecc71;

Loading