HIP-1193: Records to block streams cutover (#1381)

Signed-off-by: Michael Garber <michael.garber@hashgraph.com>

Author: mgarbs

HIP/hip-1193.md

@@ -0,0 +1,266 @@
+---
+hip: 1193 
+title: Record Stream to Block Stream Cutover
+author: Mark Blackman <mark@hashgraph.com>
+working-group: Steven Sheehy <@steven-sheehy>, Joseph Sinclair<@jsync-swirlds>, Dan Alvizu <dan.alvizu@swirldslabs.com>, Jasper Potts <jasper.potts@swirldslabs.com>, Keith Kowal <keith.kowal@swirldslabs.com>
+requested-by: Hashgraph
+type: Standards Track
+category: Mirror
+status: Review
+discussions-to: https://github.com/hiero-ledger/hiero-improvement-proposals/pull/1193
+needs-hiero-approval: Yes
+needs-hedera-review: Yes
+hedera-review-date:
+hedera-acceptance-decision:
+requires: 1056
+replaces: 679
+created: 2025-05-08
+updated: 2026-01-21
+---
+
+## Abstract
+This HIP defines the requirements and implementation details for transitioning the Hedera network from record and event streams to block streams. This transition is a critical step toward enhancing blockchain compatibility, improving data integrity, and enabling future network optimizations such as state proofs. The proposal outlines the coordinated changes required across consensus nodes, mirror nodes, and supporting infrastructure to ensure a clean cutover without service disruption. It specifies new storage paths, file formats, and mechanisms to handle the transition state, including a marker file approach to indicate the final record stream entry before blocks begin.
+
+## Motivation
+As highlighted in the Block Streams HIP:1056, the Hedera network currently produces record streams as its primary method of exposing transaction data to mirror nodes and external consumers. While functional, this approach diverges from standard blockchain architectures that organize data into blocks. By transitioning to block streams, Hedera will:
+
+1. Improve blockchain compatibility and interoperability with existing blockchain tools and infrastructure  
+2. Provide better data integrity through block-level proofs and signatures  
+3. Reduce storage costs through optimized block organization and eventual deduplication  
+4. Enable future improvements like block-based state proofs and block node architecture  
+5. Simplify the developer experience by aligning with familiar blockchain concepts
+
+This transition is a prerequisite for the planned block node architecture and represents an important step in Hedera's technical evolution.
+
+## Rationale
+The transition from record streams to block streams requires careful planning to ensure network continuity, data integrity, and backward compatibility. This HIP proposes a clean cutover approach with the following key design decisions:
+
+### Bucket Structure and File Format
+
+The new block stream files will follow a revised path structure that includes network, shard, realm, and node ID components:
+
+```
+block/{network}-{YYYYMMDD_HHmm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz
+```
+
+This structure provides:
+
+- Clear separation from existing record streams  
+- Support for sharding  
+- Node identification to identify source of block  
+- Block number-based naming for sequential processing  
+- ISO 8601 date and time format values of last network reset (optional field for resettable networks)
+
+Block files will use the `.blk.gz` extension and GZip compression to maintain better compatibility with current record stream processing.
+
+### Cutover Mechanism
+
+A marker file will be used to denote the last record stream entry, providing mirror nodes with a clear signal to transition processing from records to blocks. This approach:
+
+- Creates a clean demarcation point between systems  
+- Allows mirror nodes to detect the transition without configuration changes  
+- Prevents data loss during the transition  
+- Eases mirror node operations, enabling mirror nodes to update mirror node version before block stream cutover and have seamless service continuity.   
+- Reduces mirror node costs by not requiring periodically polling for block files.
+
+### Node Requirements
+
+Consensus nodes will:
+
+- Stop producing record and event streams at the designated cutover point  
+- Generate marker files to signal the end of record streams  
+- Begin producing block streams with proper continuity from record stream hash to block hash.
+- Update uploaders to use new block paths
+
+Mirror nodes will:
+
+- Implement logic to detect marker files and switch processing modes  
+- Support the new block path structure  
+- Implement block proof verification using Threshold Signature Scheme (TSS)  
+- Support retry mechanisms for failed block ingestion
+
+## User stories
+1. As a mirror node operator, I want a clear signal when to transition from record stream processing to block stream processing, so that I don't miss any transactions during the cutover.  
+     
+2. As a mirror node operator, I want to understand the new block file format and paths so that I can adjust my infrastructure to process block streams efficiently.  
+     
+3. As a developer building on Hedera, I want to understand how the transition affects data availability and processing so that my applications continue to function correctly.  
+     
+4. As a consensus node operator, I want to know exactly what changes are needed to support block streams so that I can prepare for the upgrade.
+
+5. As a mirror node operator I need to know what version of the mirror node software I need to be running and have sufficient time to upgrade to that version before the cutover  
+     
+6. As a network user, I want the transition to be seamless with no impact on transaction processing or data availability.
+
+7. As a consumer of mirror node data and associated metrics, I want the transition to be seamless with no impact on how I consume mirror node data.
+8. As a consensus node operator, I want to notify stream consumers with an end of record streams marker to signify the last record stream file during an upgrade.
+## Specification
+### 1\. Cutover Point Definition
+
+The transition from record streams to block streams will occur during a Hedera mainnet update  boundary, with the following requirements:
+
+- Occurs after a Hedera Freeze Upgrade transaction during Hedera mainnet update window  
+- Current round comes to consensus  
+- The final record stream will be completed and a marker file will be generated  
+- The first block will contain zero transactions to ensure proper formatting without impacting network usage  
+- Block continuity will be maintained by carrying forward:  
+  - The correct block number (incremented by one from the last record)  
+  - The last running hash value &mdash; cryptographic continuity of the Hedera blockchain.
+
+### 2\. Consensus Node Requirements
+
+#### 2.1 Record and Event Stream Termination
+
+Consensus nodes will:
+
+- Perform Freeze operation and stop accepting transactions  
+- Freeze round comes to consensus  
+- Ensure all signatures are collected and files are properly finalized  
+- Produce final record file  
+- Stop producing both record and event streams at the designated cutover point  
+- Generate marker files to indicate the end of record and event streams
+
+#### 2.2 Block Stream Initialization
+
+Consensus nodes will:
+
+- Construct the first block with zero transactions
+- Apply the correct running hash and block number from the final record  
+- Upload to new block path structure for uploaders
+
+NOTE: All event stream format changes must be completed and active prior to the cutover
+
+#### 2.3 Path Structure Implementation
+
+Consensus nodes will implement the new path structure, for example:
+
+```
+block/mainnet/0/0/10/000000000000000000000000000083406349.blk.gz
+```
+
+- The network value will identify the network (mainnet, testnet, etc.)  
+- The shard and realm values will be included in the directory structure  
+- The nodeID will identify the consensus node that produced the block  
+- Block files will use a 36-digit zero-padded block number format  
+- Files will use gzip compression and the `.blk.gz` extension
+
+### 3\. Mirror Node Requirements
+
+#### 3.1 Processing Logic
+
+Mirror nodes will implement logic to:
+
+- Detect marker files indicating the end of record streams  
+- Switch from record stream processing to block stream processing  
+- Apply the new bucket directory structure and path conventions  
+- Store block data in a compatible format with existing database schemas
+
+#### 3.2 Block Verification
+
+Mirror nodes will:
+
+- Implement block proof verification using Threshold Signature Scheme (TSS)   
+- Implement retry mechanisms to handle cases where block verification fails.  Should retry the same block number from other node until correctly processed.
+
+#### 3.3 Bootstrap and Recovery
+
+Mirror nodes will:
+
+- Implement logic to process both block and records files.  
+- Implement logic to update the timestamp-based startup logic to use blocks rather than timestamps   
+- Take PostgreSQL snapshots before the transition to enable mirror node bootstrapping  
+- Support recovery in cases where block processing fails
+
+### 4\. Test Plan and Verification
+
+The end-to-end test plan will include:
+
+1. Verification of logic for writing block streams  
+2. Verification of block production and correctness  
+3. Verification of ingestion and cloud storage  
+4. Verification of block proof creation and verification following node upgrade  
+5. Verification of Mirror Node transition detection and cutover capability
+
+### 5\. Disaster Recovery Considerations
+
+The following risk scenarios must be addressed:
+
+#### 5.1 Pre-First-Block Risks
+
+- Node failure during cutover  
+- Uploader misconfiguration (incorrect path, cloud provider, etc.)  
+- No blocks being produced  
+- Record or event streams not stopping as intended
+
+#### 5.2 First Block Production Risks
+
+- Incorrect running hash  
+- Incorrect block number  
+- Malformed block data or schema  
+- Invalid block proof  
+- Node reconnection issues after maintenance window
+
+### Example Specification
+NA
+
+## Backwards Compatibility
+This transition introduces a fundamental change in how transaction data is exposed by the Hedera network.
+
+To maintain backward compatibility:
+
+Historical record stream data will remain available in its current format and location
+Mirror nodes will support both record stream and block stream processing
+The first block will maintain cryptographic continuity with the last record
+
+Long-term, as the ecosystem transitions to block nodes, historic record streams and cloud bucket access may eventually be deprecated, but this would require a separate HIP and migration plan.
+
+## Security Implications
+The transition to block streams introduces several security considerations:
+
+1. **Data Integrity**: Ensuring the running hash continuity between record streams and block streams is critical to maintain the cryptographic integrity of the ledger.  
+     
+2. **Block Verification**: Mirror nodes must properly implement the TSS verifier to validate block proofs.  
+     
+3. **Disaster Recovery**: Proper procedures must be in place to handle node failures or other issues during the cutover.  
+     
+4. **Data Availability**: Ensuring no transactions are lost during the transition is critical for maintaining network trust.
+
+## How to Teach This
+The transition from record streams to block streams represents a significant architectural change for the Hedera network. To help the community understand this change:
+
+1. **For Mirror Node Operators:**  
+     
+   - Provide detailed documentation on the new block file format and paths  
+   - Explain how to detect the transition marker and switch processing modes  
+   - Offer guidance on handling the transition period
+
+2. **For Developers:**  
+     
+   - Explain the conceptual shift from records to blocks  
+   - Document how transaction data will be organized in the new format  
+   - Provide examples of how to process block data
+
+3. **For Node Operators:**  
+     
+   - Detail the changes required to consensus node configuration  
+   - Explain the new uploader paths and configurations  
+   - Provide a timeline and checklist for the upgrade
+
+## Reference Implementation
+NA
+
+## Rejected Ideas
+None to note
+
+## Open Issues
+Block Stream Protobuf Definitions: The block stream protobuf definitions are still under development. Several open items remain.
+
+Block Deduplication Strategy: A decision is needed on whether to implement deduplication of blocks from multiple nodes before block nodes are fully operational, or to accept the additional storage costs during the transition period.
+
+Timestamp to Block Number Mapping: A solution is needed for mirror nodes to map between timestamps and block numbers, particularly for startup and historical data access.
+## References
+A collection of URLs used as references throughout the HIP.
+
+## Copyright/license
+This document is licensed under the Apache License, Version 2.0 —
+see [LICENSE](../LICENSE) or <https://www.apache.org/licenses/LICENSE-2.0>.