|
| 1 | +--- |
| 2 | +title: Data Lake Integration |
| 3 | +sidebar_position: 65 |
| 4 | +--- |
| 5 | + |
| 6 | +Expand your RPC node's capabilities by connecting it to a data lake for complete historical ledger access. |
| 7 | + |
| 8 | +#### Integration Overview |
| 9 | + |
| 10 | +RPC version 23.0 introduces data lake integration for the `getLedgers` endpoint, enabling access to the historical ledgers outside your node's local retention period (typically 7 days). All other RPC endpoints will still operate based on the `HISTORY_RETENTION_WINDOW` configured on your node. |
| 11 | + |
| 12 | +The process involves setting up a ledger data lake and then configuring your RPC node to use it. |
| 13 | + |
| 14 | +### 1. Accessing a Data Lake |
| 15 | + |
| 16 | +You have two options for utilizing a data lake: |
| 17 | + |
| 18 | +- **Public Data Lake:** The simplest way is to use a publicly available data lake. For example, Stellar ledger data lake is available through the AWS Open Data program at `s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet`. |
| 19 | +- **Self-Hosted Data Lake:** This method lets you have more control over data integrity, availability and access, but requires you to create and manage your own data lake. The **Galexie** tool can help you deploy a data lake on either AWS S3 or Google Cloud Storage (GCS). For detailed instructions, refer to the [Galexie Admin Guide](../../../../data/indexers/build-your-own/galexie/README.mdx). |
| 20 | + |
| 21 | +### 2. Configuring RPC for Data Lake Integration |
| 22 | + |
| 23 | +#### Pre-requisite |
| 24 | + |
| 25 | +Before you begin, configure your RPC node with cloud provider credentials and ensure it has read permissions for the data lake bucket. |
| 26 | + |
| 27 | +#### Configuration Steps |
| 28 | + |
| 29 | +Update your RPC node's configuration file with the following settings: |
| 30 | + |
| 31 | +1. **Specify Storage Path:** Define the storage backend (`GCS` or `S3`) and provide the full path to the bucket (e.g., `my-bucket/path/to/data`). |
| 32 | +2. **Enable the Feature Flag:** Set `SERVE_LEDGERS_FROM_DATASTORE` to `true`. |
| 33 | +3. **Configure Retention Window:** The `HISTORY_RETENTION_WINDOW` must be a non-zero value, even though data is fetched from the datastore. Ledgers within this retention window are served from the RPC’s local storage, while older ledgers are retrieved from the data lake. A minimum value of `1` is sufficient. |
| 34 | + |
| 35 | +#### Configuration Examples |
| 36 | + |
| 37 | +Below are examples for configuring GCS and S3 backends. |
| 38 | + |
| 39 | +A. GCS Configuration Example |
| 40 | + |
| 41 | +```toml |
| 42 | +# External datastore configuration for GCS |
| 43 | +[datastore_config] |
| 44 | + type = "GCS" |
| 45 | + |
| 46 | +[datastore_config.params] |
| 47 | + destination_bucket_path = "your-bucket/path/to/data" |
| 48 | + |
| 49 | +[datastore_config.schema] |
| 50 | + ledgers_per_file = 1 |
| 51 | + files_per_partition = 64000 |
| 52 | + |
| 53 | +# Enable fetching historical ledgers from the datastore when not available locally |
| 54 | +SERVE_LEDGERS_FROM_DATASTORE = true |
| 55 | +``` |
| 56 | + |
| 57 | +B. S3 Configuration Example |
| 58 | + |
| 59 | +```toml |
| 60 | +# External datastore configuration for S3 |
| 61 | +[datastore_config] |
| 62 | + type = "S3" |
| 63 | + |
| 64 | +[datastore_config.params] |
| 65 | + destination_bucket_path = "your-bucket/path/to/data`" |
| 66 | + region = "your_s3_region" # e.g., "us-east-1" |
| 67 | + |
| 68 | +[datastore_config.schema] |
| 69 | + ledgers_per_file = 1 |
| 70 | + files_per_partition = 64000 |
| 71 | + |
| 72 | +# Enable fetching historical ledgers from the datastore when not available locally |
| 73 | +SERVE_LEDGERS_FROM_DATASTORE = true |
| 74 | +``` |
| 75 | + |
| 76 | +### 3. Verifying the Setup |
| 77 | + |
| 78 | +After configuring your RPC node, you can verify that the integration is working by making a `GetLedgers` request for a ledger sequence number that's older than your node’s standard retention window.The RPC should successfully return the ledger data from the data lake. |
| 79 | + |
| 80 | +Example Request: |
| 81 | + |
| 82 | +``` |
| 83 | +curl -X POST https://<rpc-host>/ \ |
| 84 | +-H "Content-Type: application/json" \ |
| 85 | +-d '{ |
| 86 | + "jsonrpc": "2.0", |
| 87 | + "id": "1", |
| 88 | + "method": "getLedgers", |
| 89 | + "params": { |
| 90 | + "startLedger": 100, |
| 91 | + "pagination": { |
| 92 | + "limit": 1 |
| 93 | + } |
| 94 | + } |
| 95 | +}' |
| 96 | +``` |
0 commit comments