[tracking] CRC (version checksum) file support

[scottsand-db]

Summary

Tracking support for reading and using CRC (version checksum) files to optimize log replay operations.

A CRC file contains a snapshot of table state at a specific version (protocol, metadata, ICT, domain metadata, statistics, etc.), enabling faster P&M replay by avoiding full log replay when possible.

Status: ✅ = Done, 🛠️ = WIP/Needs Review, 🟠 = TODO

Progress

Reading CRC

Owner	Description	PR	Created	Merged	Status
scottsand-db	Extract P&M replay into log_segment/protocol_metadata_replay.rs	#1782	02/05/26	02/06/26	✅
scottsand-db	CRC module: schema, visitor, reader, lazy loader	#1780	02/05/26	02/09/26	✅
scottsand-db	Fix existing CRC test to use correct JSON format	#1789	02/06/26		🛠️
scottsand-db	Use CRC in P&M replay	#1790	02/06/26	02/12/26	✅
scottsand-db	Use CRC for ICT	#1806	02/10/26	02/24/26	✅
scottsand-db	Add internal API for retrieving file stats from CRC	#1944	02/24/26	02/27/26	✅
scottsand-db	Refactor CRC reader from json_handler+visitor to serde_json	#1963	02/26/26	02/28/26	✅
scottsand-db	Read domain_metadata from CRC files via serde	#1968	02/26/26	02/28/26	✅
scottsand-db	Store CRC domain_metadata as HashMap	#1991	03/02/26	03/03/26	✅
scottsand-db	Remove LazyCrc from Snapshot::new internal-api signature	#2002	03/03/26	03/03/26	✅
Jameson-Crate	Read/parse support for set transactions from CRC	#2022	03/05/26	03/09/26	✅
scottsand-db	Ensure CRC domain metadata never updates from None to Some	#2058	03/09/26	03/11/26	✅
scottsand-db	Consolidate domain metadata reads through Snapshot	#2065	03/10/26	03/11/26	✅
scottsand-db	Use CRC for domain metadata reads	#2104	03/12/26	03/12/26	✅
scottsand-db	CRC replay visitor and ascending CRC replay engine	#2007	03/04/26		🛠️
	Extract txn_id from CRC				🟠
	Extract file_size_histogram from CRC				🟠
	Extract all_files from CRC				🟠
	Extract num_deleted_records_opt from CRC				🟠
	Extract num_deletion_vectors_opt from CRC				🟠
	Extract deleted_record_counts_histogram_opt from CRC				🟠

Writing CRC

Owner	Description	PR	Created	Merged	Status
scottsand-db	Add StorageHandler::put and CRC file writing	#1984	02/28/26	03/02/26	✅
scottsand-db	FileStatsDelta computation	#1994	03/02/26	03/03/26	✅
scottsand-db	Add CrcDelta and Crc::apply for incremental CRC state tracking	#2005	03/04/26	03/06/26	✅
scottsand-db	Expose CRC getter behind test-utils feature flag	#2041	03/06/26	03/06/26	✅
scottsand-db	Make CRC file stats private with safe accessor	#2045	03/08/26	03/09/26	✅
scottsand-db	Inject CrcDelta into post-commit snapshot	#2028	03/06/26	03/09/26	✅
scottsand-db	Support CRC writes (end-to-end)	#2050	03/09/26	03/12/26	✅

Incremental CRC

Focused subseries of PRs (stack/incremental_crc_*) refactoring the CRC system toward typed state enums and a reverse-replay accumulator.

Owner	Description	PR	Created	Merged	Status
scottsand-db	incremental_crc_1: introduce CrcRaw serde intermediate, drop num_metadata/num_protocol from in-memory Crc	#2559	05/13/26	05/14/26	✅
scottsand-db	incremental_crc_2: introduce FileStatsState typed enum + custom Serialize that rejects degraded states	#2562	05/14/26	05/15/26	✅
scottsand-db	incremental_crc_3: introduce DomainMetadataState typed enum	#2567	05/14/26	05/15/26	✅
scottsand-db	incremental_crc_4: introduce SetTransactionState typed enum	#2570	05/15/26	05/20/26	✅
scottsand-db	incremental_crc_5a: reshape CrcDelta for HashMap-keyed DM/txn + single is_incremental_safe	#2583	05/20/26	05/21/26	✅
scottsand-db	incremental_crc_5b: enforce ICT enablement consistency on write	#2584	05/20/26	05/21/26	✅
scottsand-db	incremental_crc_5c-pre: add version field to Crc + two tiny CRC refactors	#2597	05/22/26	05/22/26	✅
scottsand-db	incremental_crc_5c-main: CrcReplayAccumulator + visitor for stale-CRC reverse replay	#2602	05/22/26	05/27/26	✅
scottsand-db	crc_tests_refactor: composable action DSL for CrcReadTest builder	#2616	05/26/26	05/26/26	✅
scottsand-db	incremental_crc_5c-tests: integration test suite for incremental CRC replay	#2614	05/26/26	05/29/26	✅
scottsand-db	incremental_crc_6a: extract incremental update to snapshot/incremental.rs + section headers	#2647	05/28/26	05/28/26	✅
scottsand-db	incremental_crc_6b: replace LazyCrc with eager Option<Arc<Crc>>	#2655	05/29/26	06/01/26	✅
scottsand-db	incremental_crc_6c: advance stale CRC via reverse-replay on snapshot load + delete Case 2 cascade	#2678	06/02/26	06/08/26	✅
scottsand-db	Advance existing CRC over new tail commits on incremental snapshot update (builder_from) instead of dropping it (#2674)	#2712	06/08/26		🛠️
	Add metrics test for SetTxn / Dm for Partial case				🟠
	Support log compaction files in incremental CRC reverse-replay (#2615)				🟠
	Degrade to Indeterminate on negative file size during incremental CRC replay (#2676)				🟠
	Emit an IncrementalCrcLoad metric when snapshot P&M is served from the CRC (#2677)				🟠
	Reuse the CRC read during snapshot load in write_checksum instead of re-reading from disk (#2889)				🟠
scottsand-db	Make CRC Default impls test-only (#2568)		05/14/26		🟠
	Use a stale Partial DM cache as a base for log-replay misses: only scan commits older than the cache instead of re-scanning the entire log_segment. Today Snapshot::get_domain_metadatas_internal discards the Partial cache and rescans from scratch when any filter key misses. See kernel/src/snapshot/mod.rs get_domain_metadatas_internal TODO.				🟠
	Track tombstones in DomainMetadataState::Partial (#2572) so removals observed during apply return authoritative None instead of falling through to log replay.				🟠
	Migrate the upstream commit chain (CommitMetadata.domain_metadata_changes, Transaction.set_transactions) from Vec to HashMap<String, _>, lifting protocol-mandated uniqueness from runtime checks into the type system. Drops the Vec→HashMap conversion currently sitting in Transaction::build_crc_delta (kernel/src/transaction/mod.rs).				🟠

TODO: bound DM query loading when CRC present
TODO: bound SetTxn query loading when CRC present
TODO: cache CRC again
TODO: remove ANALYZE restriction
TODO: drive forward incremental protocol change

Testing

Owner	Description	PR	Created	Merged	Status
	CRC tests: time travel scenarios				🟠
	CRC tests: log compaction scenarios				🟠
	CRC tests: build_from scenarios				🟠
	CRC tests: _last_checkpoint scenarios				🟠