Skip to content

Commit d3c407e

Browse files
jhayniffyjhayniffy
authored
Merge pull request #284 from marvs8/feat/indexer-batch-size
Feat/indexer batch size
2 parents bc1c6c0 + 898f047 commit d3c407e

3 files changed

Lines changed: 289 additions & 0 deletions

File tree

‎.kiro/specs/indexer-batch-size/.config.kiro‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"specId": "ee68c70d-ee1c-4764-b4a9-96c5676223ad", "workflowType": "requirements-first", "specType": "feature"}

‎.kiro/specs/indexer-batch-size/design.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Design Document: Indexer Batch Size
2+
3+
## Overview
4+
5+
This feature replaces the hardcoded `BATCH_SIZE = 50` in `IndexerService` with a value read from `ConfigService` on every job cycle. The batch size is validated at startup via the existing Joi schema, exposed as a Prometheus histogram for observability, and documented in an operator runbook.
6+
7+
The change is intentionally minimal: no new modules, no new database tables, no new API endpoints. The three touch-points are `env.validation.ts`, `IndexerService`, and `MetricsService`.
8+
9+
## Architecture
10+
11+
```mermaid
12+
flowchart TD
13+
ENV[".env / process.env\nINDEXER_BATCH_SIZE"] -->|Joi validation at startup| CFG[ConfigService]
14+
CFG -->|get() per cycle| IS[IndexerService\nprocessNextBatchForNetwork]
15+
IS -->|batchSize param| SRB[SorobanService\ngetEvents]
16+
IS -->|observe duration| MS[MetricsService\nindexer_batch_processing_duration_seconds]
17+
MS -->|/metrics| PROM[Prometheus scrape]
18+
```
19+
20+
Key design decision: the batch size is read via `this.config.get()` inside `processNextBatchForNetwork` on every call rather than cached in the constructor. This satisfies Requirement 5 (dynamic reconfiguration) without any additional infrastructure — NestJS `ConfigService` already reads from the validated environment object, so a process restart is the only mechanism that changes the value in practice, but the code is structured to pick up any future hot-reload mechanism automatically.
21+
22+
## Components and Interfaces
23+
24+
### env.validation.ts — new field
25+
26+
```typescript
27+
INDEXER_BATCH_SIZE: Joi.number()
28+
.integer()
29+
.min(1)
30+
.max(100)
31+
.default(10)
32+
.description('Max ledger events fetched per Soroban RPC call (1–100, default 10)'),
33+
```
34+
35+
### IndexerService — changes
36+
37+
- Remove `private readonly BATCH_SIZE = 50`
38+
- In `processNextBatchForNetwork`, read batch size inline:
39+
```typescript
40+
const batchSize = this.config.get<number>('INDEXER_BATCH_SIZE', 10);
41+
```
42+
- Wrap the `soroban.getEvents(startLedger, batchSize)` call with a timer and record to `MetricsService`.
43+
44+
### MetricsService — new metric
45+
46+
New field added in the constructor, following the existing pattern:
47+
48+
```typescript
49+
readonly indexerBatchDuration: client.Histogram<string>;
50+
```
51+
52+
Registered as:
53+
54+
```typescript
55+
this.indexerBatchDuration = new client.Histogram({
56+
name: 'indexer_batch_processing_duration_seconds',
57+
help: 'Wall-clock time to fetch and process one indexer batch',
58+
labelNames: ['network'],
59+
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30],
60+
registers: [this.registry],
61+
});
62+
```
63+
64+
A helper method `recordIndexerBatch` is added to keep call-sites clean:
65+
66+
```typescript
67+
recordIndexerBatch(network: string, durationMs: number): void {
68+
this.indexerBatchDuration.observe({ network }, durationMs / 1000);
69+
}
70+
```
71+
72+
### IndexerModule — no change required
73+
74+
`MetricsModule` is already imported globally (via `AppModule`), so `MetricsService` is injectable into `IndexerService` without touching `indexer.module.ts`. If it is not globally exported, add `MetricsModule` to the `imports` array.
75+
76+
## Data Models
77+
78+
No schema changes. The feature is entirely in-process configuration and metrics.
79+
80+
The only "data" involved is the validated environment value:
81+
82+
| Variable | Type | Min | Max | Default | Source |
83+
|---|---|---|---|---|---|
84+
| `INDEXER_BATCH_SIZE` | integer | 1 | 100 | 10 | `env.validation.ts` Joi schema |
85+
86+
## Error Handling
87+
88+
| Scenario | Behaviour |
89+
|---|---|
90+
| `INDEXER_BATCH_SIZE` absent | Joi applies default `10`; startup proceeds normally |
91+
| `INDEXER_BATCH_SIZE = 0` | Joi rejects at startup with `"INDEXER_BATCH_SIZE" must be greater than or equal to 1` |
92+
| `INDEXER_BATCH_SIZE = 101` | Joi rejects at startup with `"INDEXER_BATCH_SIZE" must be less than or equal to 100` |
93+
| `INDEXER_BATCH_SIZE = "abc"` | Joi rejects at startup with `"INDEXER_BATCH_SIZE" must be a number` |
94+
| `config.get()` returns `undefined` at runtime | Fallback default `10` passed as second arg to `config.get<number>('INDEXER_BATCH_SIZE', 10)` |
95+
| Metric observe throws | Wrapped in try/catch; error is logged but does not interrupt the indexer loop |
96+
97+
Startup validation is the primary guard. Runtime fallback is a belt-and-suspenders safety net.
98+
99+
100+
## Correctness Properties
101+
102+
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
103+
104+
### Property 1: Valid batch sizes are accepted by the schema
105+
106+
*For any* integer `n` in the range `[1, 100]`, validating `{ INDEXER_BATCH_SIZE: n }` against the Joi schema should succeed and return `n` as the resolved value.
107+
108+
**Validates: Requirements 1.1**
109+
110+
---
111+
112+
### Property 2: Out-of-range integers are rejected by the schema
113+
114+
*For any* integer `n` where `n < 1` or `n > 100`, validating `{ INDEXER_BATCH_SIZE: n }` against the Joi schema should produce a validation error. This covers both the lower boundary (0, negatives) and the upper boundary (101+).
115+
116+
**Validates: Requirements 1.2, 1.3, 4.4, 4.5**
117+
118+
---
119+
120+
### Property 3: Non-integer values are rejected by the schema
121+
122+
*For any* non-integer value (float, string, boolean) supplied as `INDEXER_BATCH_SIZE`, the Joi schema should produce a validation error.
123+
124+
**Validates: Requirements 1.4**
125+
126+
---
127+
128+
### Property 4: Batch size is forwarded to soroban.getEvents on every cycle
129+
130+
*For any* valid batch size `n` returned by `ConfigService.get('INDEXER_BATCH_SIZE')`, a call to `processNextBatchForNetwork` should invoke `soroban.getEvents` with exactly `n` as the limit argument. This holds for boundary values (1, 100) and the default (10).
131+
132+
**Validates: Requirements 2.2, 4.1, 4.2, 4.3**
133+
134+
---
135+
136+
### Property 5: Batch size is re-read from ConfigService on each cycle (dynamic reconfiguration)
137+
138+
*For any* two consecutive calls to `processNextBatchForNetwork` where `ConfigService.get` returns different values `n1` and `n2`, the first call should invoke `soroban.getEvents` with `n1` and the second with `n2`. The service must not cache the value between cycles.
139+
140+
**Validates: Requirements 2.4, 5.1, 5.2**
141+
142+
---
143+
144+
### Property 6: Batch processing duration is recorded for every batch
145+
146+
*For any* call to `processNextBatchForNetwork` — whether it processes events or returns zero — `MetricsService.recordIndexerBatch` should be called exactly once with the correct `network` label and a non-negative duration in milliseconds.
147+
148+
**Validates: Requirements 3.2, 3.3**
149+
150+
---
151+
152+
## Testing Strategy
153+
154+
### Dual Testing Approach
155+
156+
Both unit tests and property-based tests are required. They are complementary:
157+
158+
- Unit tests cover specific examples, integration wiring, and edge cases.
159+
- Property tests verify universal invariants across randomised inputs.
160+
161+
### Unit Tests (specific examples and wiring)
162+
163+
- `env.validation.ts`: assert default value `10` when `INDEXER_BATCH_SIZE` is absent (Req 1.5).
164+
- `MetricsService`: assert `indexer_batch_processing_duration_seconds` is present in the Prometheus registry with label `network` after module init (Req 3.1, 3.4, 3.5).
165+
- `IndexerService` constructor: assert `ConfigService.get` is called during init (Req 2.1).
166+
167+
### Property-Based Tests
168+
169+
Library: **`fast-check`** (already a common choice in TypeScript/NestJS projects; add as a dev dependency if not present).
170+
171+
Each property test runs a minimum of **100 iterations**.
172+
173+
| Test | Arbitrary | Assertion | Design Property |
174+
|---|---|---|---|
175+
| Valid range accepted | `fc.integer({ min: 1, max: 100 })` | Schema returns value, no error | Property 1 |
176+
| Out-of-range rejected | `fc.oneof(fc.integer({ max: 0 }), fc.integer({ min: 101 }))` | Schema throws validation error | Property 2 |
177+
| Non-integer rejected | `fc.oneof(fc.float(), fc.string(), fc.boolean())` | Schema throws validation error | Property 3 |
178+
| Batch size forwarded | `fc.integer({ min: 1, max: 100 })` | `soroban.getEvents` called with that exact value | Property 4 |
179+
| Dynamic reconfiguration | `fc.tuple(fc.integer({min:1,max:100}), fc.integer({min:1,max:100}))` | Each cycle uses the value returned by ConfigService at that call | Property 5 |
180+
| Duration always recorded | `fc.record({ network: fc.string(), events: fc.array(fc.anything()) })` | `recordIndexerBatch` called once per `processNextBatchForNetwork` invocation | Property 6 |
181+
182+
Tag format for each test:
183+
```
184+
// Feature: indexer-batch-size, Property <N>: <property_text>
185+
```
186+
187+
### Test File Locations
188+
189+
- `backend/src/__tests__/indexer-batch-size.property.test.ts` — all property-based tests
190+
- `backend/src/__tests__/indexer-batch-size.test.ts` — unit/example tests

‎.kiro/specs/indexer-batch-size/requirements.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Requirements Document
2+
3+
## Introduction
4+
5+
The indexer currently uses a hardcoded `BATCH_SIZE = 50` when fetching ledger events from the Soroban RPC. During catch-up (large lag), this fixed window can cause RPC timeouts; during normal operation it may generate excessive RPC calls. This feature makes the batch size configurable via an environment variable (`INDEXER_BATCH_SIZE`), validated at startup, applied dynamically on each job cycle, and observable via a Prometheus metric for average batch processing time. An operator runbook section documents tuning guidance.
6+
7+
## Glossary
8+
9+
- **IndexerService**: The NestJS service (`IndexerService`) responsible for fetching and processing ledger events from the Soroban RPC.
10+
- **IndexerWorker**: The NestJS service (`IndexerWorker`) that drives the periodic indexer loop.
11+
- **Batch**: A single call to `soroban.getEvents()` that fetches up to `INDEXER_BATCH_SIZE` ledger events.
12+
- **Batch_Size**: The maximum number of ledger events fetched per RPC call, controlled by `INDEXER_BATCH_SIZE`.
13+
- **ConfigService**: The NestJS `ConfigService` used to read validated environment variables at runtime.
14+
- **MetricsService**: The NestJS service (`MetricsService`) that registers and exposes Prometheus metrics.
15+
- **Env_Validator**: The Joi validation schema in `backend/src/config/env.validation.ts`.
16+
- **Job_Cycle**: One execution of the indexer loop — from reading the cursor to advancing it after processing a batch.
17+
- **Indexer_Lag**: The difference in ledger numbers between the chain head and the last processed ledger.
18+
- **RPC_Rate_Limit**: The request-per-second ceiling imposed by the Soroban RPC provider.
19+
20+
## Requirements
21+
22+
### Requirement 1: Environment Variable Declaration and Validation
23+
24+
**User Story:** As an operator, I want `INDEXER_BATCH_SIZE` validated at startup, so that misconfigured values are caught before the indexer runs.
25+
26+
#### Acceptance Criteria
27+
28+
1. THE Env_Validator SHALL accept `INDEXER_BATCH_SIZE` as an optional integer environment variable with a default value of `10`.
29+
2. THE Env_Validator SHALL reject values of `INDEXER_BATCH_SIZE` less than `1` with a descriptive validation error at application startup.
30+
3. THE Env_Validator SHALL reject values of `INDEXER_BATCH_SIZE` greater than `100` with a descriptive validation error at application startup.
31+
4. THE Env_Validator SHALL reject non-integer values of `INDEXER_BATCH_SIZE` with a descriptive validation error at application startup.
32+
5. WHEN `INDEXER_BATCH_SIZE` is absent from the environment, THE Env_Validator SHALL apply the default value of `10`.
33+
34+
---
35+
36+
### Requirement 2: Batch Size Applied to Ledger Fetch Loop
37+
38+
**User Story:** As an operator, I want the indexer to use the configured batch size when fetching events, so that I can tune RPC call frequency and payload size without redeploying.
39+
40+
#### Acceptance Criteria
41+
42+
1. WHEN `IndexerService` initialises, THE IndexerService SHALL read `INDEXER_BATCH_SIZE` from `ConfigService` and store it as the effective Batch_Size.
43+
2. WHEN `IndexerService.processNextBatchForNetwork` is called, THE IndexerService SHALL pass the effective Batch_Size to `soroban.getEvents()`.
44+
3. THE IndexerService SHALL remove the hardcoded `private readonly BATCH_SIZE = 50` field.
45+
4. WHEN `INDEXER_BATCH_SIZE` is set to a valid value, THE IndexerService SHALL use that value on the next Job_Cycle without requiring a process restart.
46+
47+
---
48+
49+
### Requirement 3: Batch Processing Time Metric
50+
51+
**User Story:** As an operator, I want a Prometheus metric for batch processing time, so that I can observe the impact of different batch sizes and make informed tuning decisions.
52+
53+
#### Acceptance Criteria
54+
55+
1. THE MetricsService SHALL register a Prometheus Histogram named `indexer_batch_processing_duration_seconds` with label `network`.
56+
2. WHEN a batch completes successfully, THE IndexerService SHALL record the elapsed wall-clock time of that batch in the `indexer_batch_processing_duration_seconds` histogram.
57+
3. WHEN a batch results in zero processed events, THE IndexerService SHALL still record the elapsed time in the `indexer_batch_processing_duration_seconds` histogram.
58+
4. THE MetricsService SHALL expose `indexer_batch_processing_duration_seconds` via the existing `/metrics` Prometheus scrape endpoint.
59+
5. WHEN the Prometheus scrape endpoint is queried, THE MetricsService SHALL include `indexer_batch_processing_duration_seconds` in the response body.
60+
61+
---
62+
63+
### Requirement 4: Batch Size Boundary Tests
64+
65+
**User Story:** As a developer, I want automated tests that verify batch size boundaries are respected, so that regressions are caught in CI.
66+
67+
#### Acceptance Criteria
68+
69+
1. WHEN `INDEXER_BATCH_SIZE` is set to `1`, THE IndexerService SHALL pass `1` to `soroban.getEvents()`.
70+
2. WHEN `INDEXER_BATCH_SIZE` is set to `100`, THE IndexerService SHALL pass `100` to `soroban.getEvents()`.
71+
3. WHEN `INDEXER_BATCH_SIZE` is set to `10` (the default), THE IndexerService SHALL pass `10` to `soroban.getEvents()`.
72+
4. WHEN `INDEXER_BATCH_SIZE` is set to `0` in the Env_Validator schema, THE Env_Validator SHALL produce a validation error.
73+
5. WHEN `INDEXER_BATCH_SIZE` is set to `101` in the Env_Validator schema, THE Env_Validator SHALL produce a validation error.
74+
75+
---
76+
77+
### Requirement 5: Dynamic Reconfiguration Without Restart
78+
79+
**User Story:** As an operator, I want batch size changes to take effect on the next job cycle, so that I can tune the indexer without downtime.
80+
81+
#### Acceptance Criteria
82+
83+
1. WHEN `INDEXER_BATCH_SIZE` is updated in the environment and the process is not restarted, THE IndexerService SHALL apply the updated value on the next Job_Cycle where `ConfigService` reflects the change.
84+
2. WHILE the indexer loop is running, THE IndexerService SHALL read Batch_Size from `ConfigService` on each call to `processNextBatchForNetwork` rather than caching it as a constructor-time constant.
85+
86+
---
87+
88+
### Requirement 6: Runbook Documentation
89+
90+
**User Story:** As an operator, I want a runbook section that explains how to tune `INDEXER_BATCH_SIZE` based on observed metrics, so that I can resolve indexer lag or RPC rate-limit issues without guessing.
91+
92+
#### Acceptance Criteria
93+
94+
1. THE Runbook SHALL document the valid range (`1`–`100`) and default value (`10`) of `INDEXER_BATCH_SIZE`.
95+
2. THE Runbook SHALL describe the relationship between Batch_Size and Indexer_Lag: larger batches reduce lag faster but increase per-call RPC cost.
96+
3. THE Runbook SHALL describe the relationship between Batch_Size and RPC_Rate_Limit: larger batches consume more quota per call and may trigger throttling.
97+
4. THE Runbook SHALL include a decision tree that guides operators to increase Batch_Size when Indexer_Lag is high and decrease Batch_Size when RPC errors or timeouts are observed.
98+
5. THE Runbook SHALL reference the `indexer_batch_processing_duration_seconds` metric as the primary signal for evaluating tuning decisions.

0 commit comments

Comments
 (0)