Open
Conversation
Introduces subcoin-utxo-storage crate that provides O(1) UTXO operations bypassing Substrate's Merkle Patricia Trie overhead: - NativeUtxoStorage: RocksDB-based UTXO storage with column families - MuHash commitment: Rolling hash updated O(1) per UTXO change - BlockUndo: Per-block undo data for chain reorganizations - Coin serialization compatible with Bitcoin Core's gettxoutsetinfo Also adds serialize/deserialize methods to MuHash3072 for persistence. This is Phase 1 of the native UTXO storage implementation plan, which aims to achieve 20-100x speedup for IBD at higher block heights. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This enables O(1) UTXO lookups during block verification and import by using native RocksDB storage with MuHash commitment tracking. Changes: - Add NativeUtxoStorage to BlockVerifier for fast UTXO lookups - Add NativeUtxoStorage to BitcoinBlockImporter for block updates - Add --native-utxo CLI flag to run and import-blocks commands - Wire up native storage initialization in service layer When --native-utxo is enabled: - UTXO lookups during verification use O(1) RocksDB reads - Block import updates native storage atomically - MuHash commitment is maintained for UTXO set verification 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix UtxoNotFound error when tx spends output created earlier in same block by tracking in-block UTXOs in a HashMap before batch commit - Add height sync validation to ensure native storage matches chain height - Add test case for in-block UTXO spending scenario 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This eliminates the Merkle Patricia Trie overhead for UTXO operations, achieving 20-100x speedup at scale. Block execution now shows drain_changes_ms=0 since there's no trie state to commit. Changes: - Remove Coins<T> StorageDoubleMap and CoinsCount from pallet-bitcoin - Remove CoinStorageKey trait entirely - Make NativeUtxoStorage mandatory (not optional) - Update all crates to use native storage as the only UTXO source - Stub out dumptxoutset/gettxoutsetinfo (TODO: implement with native iter) Performance at block ~197k: - execute_block_ms: 0-7ms (proportional to tx count) - drain_changes_ms: 0ms (no trie updates) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement two-phase block verification: - Phase 1 (Sequential): UTXO validation, double-spend checks, task collection - Phase 2 (Parallel): Script verification via rayon par_iter Key changes: - Add ScriptVerificationTask struct for independent verification - Add verify_scripts_parallel() and verify_scripts_sequential() functions - Add --no-parallel-verification CLI flag for A/B benchmarking - Add verify_scripts_ms timing to block import logs Benchmark results (blocks 300k-320k): - Script verification: 7.2x faster (414s → 58s for 10k blocks) - Overall block import: 1.48x faster (20.4min → 13.8min) - Throughput: 8.2 bps → 12.1 bps 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Rename the crate and struct to better align with Substrate terminology: - Crate: subcoin-utxo-storage → subcoin-bitcoin-state - Struct: NativeUtxoStorage → BitcoinState - Variables: native_utxo_storage/utxo_storage → bitcoin_state - Path: "native_utxo" → "bitcoin_state" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Storage layer extensions for P2P UTXO fast sync: - bulk_import(): Import UTXOs in bulk without undo data - export_chunk(): Export UTXOs in paginated chunks - finalize_import(): Set height after sync completion - clear(): Reset storage for sync restart - verify_muhash(): Verify against checkpoint - iter_utxos(): Iterator for debugging E2E tests simulating two-node sync with mock protocol: - Basic sync between server and client nodes - Large UTXO set sync with chunked pagination - MuHash mismatch detection - Sync interruption and restart - Concurrent export/import - Chunk boundary handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add checkpoints.json with MuHash/UTXO count data from Bitcoin Core v29.99.0 - Load checkpoints at compile time with full metadata (height, block_hash, txouts, muhash) - Implement UTXO sync protocol messages (GetUtxoSetInfo, GetUtxoChunk, responses) - Add UtxoSyncStrategy for downloading UTXOs from peers with cursor-based pagination - Extend BitcoinState with bulk_import, export_chunk, finalize_import methods - Add MuHash verification at node startup via verify_and_repair_muhash() - Fix BIP30 duplicate coinbase handling (blocks 91722/91812, 91880/91842) - Add integration tests for count consistency and UTXO sync flow Checkpoint data verified against Bitcoin Core at heights: 0, 100K, 200K, 300K, 400K, 500K, 600K, 700K, 750K, 800K, 840K 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename UtxoSyncStrategy -> UtxoSnapSync (clearer name) - Rename UtxoSyncProgress -> UtxoSnapSyncProgress - Rename UTXO_SYNC_STRATEGY_KEY -> UTXO_SNAP_SYNC_KEY - Add comprehensive module documentation with protocol flow diagram - Document verification guarantees and error handling - Update fast-sync-implementation-plan.md with current status 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move `verify_script_task` function to `ScriptVerificationTask::verify` method for more idiomatic Rust and better encapsulation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move Coin methods to subcoin-runtime-primitives with #[cfg(feature = "std")] - from_txout(), encode_for_storage(), decode_from_storage(), serialize_for_muhash() - Remove duplicate Coin definition from subcoin-bitcoin-state - Simplify coin.rs to re-export Coin and keep utility functions - Add ScriptEngine::is_enabled() for clearer script verification checks - Make verify_scripts() a method of BlockVerifier - Export ScriptVerification enum from sc-consensus-nakamoto - Simplify find_utxo_in_state() since both crates now use same Coin type 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
liuchengxu
force-pushed
the
native-utxo-storage
branch
from
3cfb1ee to
61b0cb6
Compare
- Move HashMap, key_to_outpoint, outpoint_to_key imports to top of storage.rs - Move Amount, ScriptBuf imports to top of runtime-primitives with #[cfg(feature = "std")] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…r handling - Move rocksdb dependency to workspace Cargo.toml - Merge coin.rs into storage.rs (outpoint_to_key, key_to_outpoint) - Re-export Coin from subcoin_runtime_primitives in lib.rs - Unify Serialization/Deserialization error variants into Bincode with #[from] - Remove redundant .map_err() calls where #[from] handles conversion - Move CHECKPOINTS static to top of checkpoints.rs and inline json_str 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove native_utxo CLI flag (Substrate UTXO state no longer exists) - Fix parallel_verification -> script_verification field name - Use ScriptVerification enum instead of bool 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Standardize on "snap sync" terminology throughout the codebase: - Rename FastSyncConfig → SnapSyncConfig - Rename --fast-sync → --snap-sync CLI flags - Rename PausedUntilFastSync → PausedUntilSnapSync - Update all related doc comments and variable names This aligns with the existing UtxoSnapSync naming convention. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces Substrate-based UTXO state with native RocksDB storage and implements P2P UTXO fast sync for rapid node bootstrapping.
Key Changes
BitcoinState(RocksDB) for UTXO setUtxoSnapSyncdownloads UTXO snapshots from peers with cursor-based paginationCommits
Testing
Test plan
cargo test -p subcoin-bitcoin-statecargo test -p subcoin-test-service🤖 Generated with Claude Code