017 blockchain module split

.github/copilot-instructions.md

@@ -49,9 +49,9 @@ C++20 (`-std=c++20`): Follow standard conventions
 - Do not include task numbers (e.g. T001, T010) in code comments. Comments should describe *what* or *why*, not reference planning artifacts.
 
 ## Recent Changes
+- 017-blockchain-module-split: Added C++20 (`-std=c++20`) + Boost (Asio, Serialization), OpenSSL (EVP SHA-256), nlohmann/json (vendored `src/json.hpp`)
 - 016-audit-remediation: Added C++20 (`-std=c++20`) + Boost (Asio, Serialization), OpenSSL (EVP SHA-256), nlohmann/json (vendored `src/json.hpp`)
 - 015-compile-time-optimization: Added C++20 (`-std=c++20`) + Boost (Asio, Serialization, Program Options), OpenSSL, nlohmann/json (vendored), Catch2 (test)
-- 014-documentation-developer-guide: Added GitHub-Flavored Markdown (documentation-only feature; no C++ changes) + N/A (Markdown files only; Mermaid diagrams rendered natively by GitHub)
 
 
 <!-- MANUAL ADDITIONS START -->

.gitignore

@@ -84,6 +84,9 @@ tests/lifecycle_tests
 tests/lifecycle_integration_tests
 tests/rpc_integration_tests
 tests/p2p_sync_integration_tests
+tests/chain_persistence_module_tests
+tests/difficulty_engine_tests
+tests/merkle_proof_tests
 
 # Test artifacts
 *.trs

.specify/feature.json

@@ -1,3 +1,3 @@
 {
-  "feature_directory": "specs/016-audit-remediation"
+  "feature_directory": "specs/017-blockchain-module-split"
 }

docs/AUDIT.md

@@ -1,7 +1,7 @@
 # Code Audit Report
 
-**Date:** 2026-04-12  
-**Scope:** Full codebase — `src/` (5,887 lines) and `tests/` (6,752 lines)
+**Date:** 2026-04-12 (updated 2026-04-13)  
+**Scope:** Full codebase — `src/` (6,179 lines) and `tests/` (6,891 lines)
 
 ---
 
@@ -423,10 +423,38 @@ Ordered by impact and effort:
 | 11 | Stream entry lookup duplication (§3.4) | **Fixed** | Extracted shared `collectEntries` lambda in `getStreamEntries()` |
 | 12 | Test setup duplication (§7.4) | **Fixed** | `TestHelpers.hpp` with shared utilities; 9 test files updated |
 
-### Not Addressed (deferred)
+---
+
+## 10. Module Split Status (017-blockchain-module-split)
+
+**Date:** 2026-04-13
+
+Addresses §6.2 (`Blockchain.cpp` at 1,024 lines mixes concerns) and recommendation #9.
+
+`Blockchain.cpp` was split into four focused modules using composition — `Blockchain<ChunkHandler>` owns each module as a member and delegates through thin wrappers.
+
+| Module | File | Lines | Responsibility |
+|--------|------|------:|----------------|
+| `ChainPersistence<ChunkHandler>` | `src/ChainPersistence.cpp` | 379 | Chunk I/O, keys, streams, stream index, recovery, archiving |
+| `DifficultyEngine` | `src/DifficultyEngine.cpp` | 95 | Difficulty calculation, adjustment-window logic, boundary cache |
+| `MerkleProofService` | `src/MerkleProofService.cpp` | 60 | Inclusion proof generation and verification |
+| `Blockchain<ChunkHandler>` (core) | `src/Blockchain.cpp` | 624 | Block creation/mining, chain ops, delegation wrappers |
+
+All modules stay under the 400-line limit (SC-001). `Blockchain.cpp` is at 624 due to 18 delegation wrappers required by the `IBlockchain` interface.
+
+**New test suites:**
+
+| Test file | Cases | Assertions |
+|-----------|------:|-----------:|
+| `tests/chain_persistence_module_tests.cpp` | 8 | 17 |
+| `tests/difficulty_engine_tests.cpp` | 8 | 11 |
+| `tests/merkle_proof_tests.cpp` | 6 | 31 |
+
+All existing tests (344 assertions in 115 test cases) continue to pass with zero regressions. Incremental build verified — touching a single module recompiles only that `.o` file (SC-003).
+
+### Still Deferred
 
 | # | Issue | Reason |
 |---|-------|--------|
-| 9 | Split `Blockchain.cpp` into modules (§6.2) | High effort; architecture change outside audit scope |
 | 10 | Narrow `IBlockchain` interfaces (§6.4) | Medium effort; deferred to future refactoring |
 | — | RPC dispatch table (§4.6, §6.1) | Deferred; helper extraction sufficient for now |

docs/ROADMAP.md

@@ -22,6 +22,7 @@ Last updated: 2026-04-12
 | 014  | Documentation & Developer Guide | Expanded README with cross-platform build instructions and two-node quickstart, configuration reference (9 CLI flags, 30 config.json fields, TLS setup), RPC API reference (20 methods with curl examples), architecture overview with Mermaid diagrams, contributing guide |
 | 015  | Compile-Time Optimization | Shared static archive (`libblockchain_core.a`) eliminates redundant compilation of 11 core source files across 14 build targets, reducing compilation units from ~177 to ~34 (~81% reduction), clean build time from ~16 min to ~2.5 min |
 | 016  | Code Audit Remediation | Fixed 5 bugs (block count, dirty flag, merkle root, IPv6 parsing, pending pool), cached difficulty per boundary with ChunkRetainGuard, extracted shared utilities (chunkFilename, parsePeerKey, RPC helpers, send_to_peers), added TestHelpers.hpp, single-threaded io_context enforcement |
+| 017  | Blockchain Module Split | Split monolithic Blockchain.cpp (1,019 lines) into four focused modules: ChainPersistence (379 lines), DifficultyEngine (95 lines), MerkleProofService (60 lines), and slimmed Blockchain core (624 lines); composition-based ownership; zero API changes; 3 new focused test suites |
 
 ## Suggested Specs
 

specs/017-blockchain-module-split/checklists/requirements.md

@@ -0,0 +1,35 @@
+# Specification Quality Checklist: Blockchain Module Split
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-04-12
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Spec is ready for `/speckit.clarify` or `/speckit.plan`.
+- Key entities reference module names (ChainPersistence, DifficultyEngine, MerkleProofService) as domain concepts, not implementation artifacts — this is intentional as they represent the logical decomposition being specified.

specs/017-blockchain-module-split/contracts/chain-persistence.md

@@ -0,0 +1,95 @@
+# Module Interface Contract: ChainPersistence\<ChunkHandler\>
+
+**Module**: `ChainPersistence<ChunkHandler>`  
+**Header**: `src/ChainPersistence.hpp`  
+**Implementation**: `src/ChainPersistence.cpp`
+
+## Construction
+
+```
+ChainPersistence(const std::filesystem::path& blockchainPath, size_t chunkSize)
+```
+
+Stores path and chunk size as invariant configuration. No other initialization.
+
+## Public Methods
+
+### Chunk I/O
+
+```
+void saveChunk(std::vector<ChunkHandler>& chain, size_t chunkIndex)
+```
+Serializes `chain[chunkIndex]` to `chunk_NNNNNN.dat` via Boost.Serialization.
+
+```
+void loadChunk(std::vector<ChunkHandler>& chain, size_t chunkIndex)
+```
+Deserializes `chunk_NNNNNN.dat` into `chain[chunkIndex]`.
+
+```
+void freeChunk(std::vector<ChunkHandler>& chain, size_t chunkIndex,
+               const std::set<size_t>& retainedChunks)
+```
+Clears `chain[chunkIndex].blocks` if not in `retainedChunks` and not the last chunk.
+
+### Index I/O
+
+```
+void saveKeys(const std::map<std::string, std::vector<size_t>>& keyIndexMap)
+void loadKeys(std::map<std::string, std::vector<size_t>>& keyIndexMap)
+
+void saveStreams(const std::set<std::string>& streamRegistry)
+void loadStreams(std::set<std::string>& streamRegistry)
+
+void saveStreamIndex(const StreamKeyIndex& streamKeyIndex)
+void loadStreamIndex(StreamKeyIndex& streamKeyIndex)
+```
+Each serializes/deserializes the given data structure to/from the corresponding `.dat` file.
+
+### Bulk Operations
+
+```
+void saveAllChunks(std::vector<ChunkHandler>& chain,
+                   const std::map<std::string, std::vector<size_t>>& keyIndexMap,
+                   const std::set<std::string>& streamRegistry,
+                   const StreamKeyIndex& streamKeyIndex,
+                   bool& dirty)
+```
+Saves all chunks, keys, streams, and stream index. Clears `dirty` on success.
+
+```
+size_t discoverChunks()
+```
+Counts existing `chunk_NNNNNN.dat` files in `blockchainPath_`. Returns count.
+
+```
+bool validateChunk(size_t chunkIndex, const ConsensusConfig& config)
+```
+Loads and validates a chunk's block linkage and hashes. Returns true if valid.
+
+### Recovery & Archiving
+
+```
+void recoverChain(std::vector<ChunkHandler>& chain,
+                  std::map<std::string, std::vector<size_t>>& keyIndexMap,
+                  std::set<std::string>& streamRegistry,
+                  StreamKeyIndex& streamKeyIndex,
+                  size_t& totalBlockCount,
+                  size_t& chunkCount,
+                  bool& dirty,
+                  std::unordered_map<size_t, uint32_t>& difficultyCache,
+                  const ConsensusConfig& config,
+                  bool fast_startup)
+```
+Discovers chunks, validates, loads indexes (or rebuilds from blocks), sets counters.
+
+```
+void archiveChainFiles(size_t chunkCount)
+```
+Creates timestamped backup directory and moves all chain files into it.
+
+## Type Aliases
+
+```
+using StreamKeyIndex = std::map<std::string, std::map<std::string, std::vector<size_t>>>;
+```

specs/017-blockchain-module-split/contracts/difficulty-engine.md

@@ -0,0 +1,64 @@
+# Module Interface Contract: DifficultyEngine
+
+**Module**: `DifficultyEngine`  
+**Header**: `src/DifficultyEngine.hpp`  
+**Implementation**: `src/DifficultyEngine.cpp`
+
+## Construction
+
+```
+DifficultyEngine() = default;
+```
+
+Stateless — no owned fields. All data passed per call.
+
+## Public Methods
+
+```
+uint32_t calculateNewDifficulty(const ConsensusConfig& config,
+                                 size_t totalBlockCount,
+                                 uint32_t currentDifficulty,
+                                 std::function<Block(size_t)> getBlock)
+```
+Computes the next difficulty based on the most recent adjustment window.
+
+**Parameters**:
+- `config`: Consensus parameters (adjustmentWindow, targetBlockInterval, maxAdjustmentFactor, minDifficulty, maxDifficulty)
+- `totalBlockCount`: Current total blocks in chain
+- `currentDifficulty`: Current PoW difficulty
+- `getBlock`: Callback to fetch a block by index (isolates from persistence)
+
+**Returns**: New difficulty value, clamped to `[minDifficulty, maxDifficulty]`.
+
+**Behavior**:
+- If `totalBlockCount <= adjustmentWindow`: returns `currentDifficulty` unchanged
+- Computes actual vs expected time ratio over the last window
+- Applies log₂ adjustment, clamped by `maxAdjustmentFactor`
+
+---
+
+```
+uint32_t getDifficultyForHeight(size_t height,
+                                 const ConsensusConfig& config,
+                                 size_t totalBlockCount,
+                                 std::unordered_map<size_t, uint32_t>& difficultyCache,
+                                 std::function<Block(size_t)> getBlock,
+                                 std::function<void(size_t)> retainChunk)
+```
+Computes the difficulty that would apply at a given chain height, using cached values when available.
+
+**Parameters**:
+- `height`: Target chain height
+- `config`: Consensus parameters
+- `totalBlockCount`: Current total blocks
+- `difficultyCache`: Mutable reference to cache (reads existing entries, writes new)
+- `getBlock`: Callback to fetch block by index
+- `retainChunk`: Callback to retain a chunk in memory during multi-access scan
+
+**Returns**: Difficulty at the given height, clamped to `[minDifficulty, maxDifficulty]`.
+
+**Behavior**:
+- Walks adjustment boundaries from genesis to `height`
+- Checks `difficultyCache` at each boundary; uses cached value if present
+- Computes and caches missing boundaries
+- Uses `retainChunk` to avoid repeated load/free cycles

specs/017-blockchain-module-split/contracts/merkle-proof-service.md

@@ -0,0 +1,59 @@
+# Module Interface Contract: MerkleProofService
+
+**Module**: `MerkleProofService`  
+**Header**: `src/MerkleProofService.hpp`  
+**Implementation**: `src/MerkleProofService.cpp`
+
+## Construction
+
+```
+MerkleProofService() = default;
+```
+
+Stateless — no owned fields. All data passed per call.
+
+## Public Methods
+
+```
+nlohmann::json getInclusionProof(const Block& block, size_t entryIndex)
+```
+Generates a Merkle inclusion proof for a specific entry within a block.
+
+**Parameters**:
+- `block`: The block containing the entry
+- `entryIndex`: Index of the entry within `block.entries`
+
+**Returns**: JSON object with fields:
+- `blockIndex`: Block index
+- `entryIndex`: Entry index within block
+- `merkleRoot`: Block's Merkle root (hex)
+- `leafHash`: SHA-256 of serialized entry (hex)
+- `proof`: Array of `{hash, isLeft}` elements tracing from leaf to root
+
+**Errors**: Throws `std::runtime_error` if `entryIndex >= block.entries.size()`.
+
+---
+
+```
+nlohmann::json verifyInclusionProof(const Block& block,
+                                     const std::string& leafHash,
+                                     const nlohmann::json& proofArray)
+```
+Verifies a Merkle inclusion proof against a block's stored Merkle root.
+
+**Parameters**:
+- `block`: The block to verify against
+- `leafHash`: Claimed leaf hash (hex)
+- `proofArray`: JSON array of `{hash, isLeft}` proof elements
+
+**Returns**: JSON object with fields:
+- `valid`: Boolean — whether proof verifies
+- `expectedRoot`: Block's stored Merkle root
+- `computedRoot`: Root recomputed from leaf + proof path
+
+## Dependencies
+
+- `MerkleTree::computeLeafHash()` — hashes serialized entries
+- `MerkleTree::generateProof()` — builds proof path
+- `MerkleTree::verifyProof()` — recomputes and checks root
+- `boost::archive::binary_oarchive` — serializes StreamEntry for leaf hashing