pr_description_escrow_pipeline.md

feat(core): rebuild event-sourced escrow pipeline with circuit-breaker RPC and exactly-once indexer

Summary

This rebuilds the core escrow data pipeline from scratch as a cohesive, production-grade system spanning four layers:

• On-chain storage (contracts/escrow_contract/src/storage.rs) — Soroban persistent storage helpers with TTL bump-on-read.
• Deterministic state machine (backend/lib/escrowStateMachine.js) — a pure, side-effect free transition table with history validation.
• Resilient RPC client (backend/services/stellarService.js) — exponential-backoff polling, circuit breaker, and 4096-ledger fan-out.
• CQRS write model (backend/services/escrowService.js) — serializable command handlers with a post-commit domain-event relay and compensation log.
• Exactly-once indexer (backend/services/escrowIndexer.js) — crash-safe cursor, idempotent upserts, and a dead-letter queue.

Every layer has full test coverage. The rebuild is not a straight restore: each module was reimplemented to the architectural requirements in the issue (TTL-on-read, 100% branch coverage, circuit breaker, Serializable isolation + double-spend prevention, idempotent upserts + DLQ).

---

contracts/escrow_contract/src/storage.rs

Design decisions

• Persistent storage only. Every protocol key lives in env.storage().persistent() so it survives instance-TTL recycling. Instance storage is reserved for the StorageManager upgrade/version shim so legacy lib.rs call sites still compile.
• TTL is bumped on every read, not just writes. Soroban evicts persistent entries once their live-until ledger lapses. A key that is read often but never written (treasury, fee bps, arbiter registry, escrow rows) would eventually be evicted; get_persistent calls extend_ttl(key, LEDGER_THRESHOLD, LEDGER_TO_LIVE) whenever a value is present, keeping hot keys alive without requiring a manual rent transaction.
• Single TTL policy source. LEDGER_THRESHOLD = 100 / LEDGER_TO_LIVE = 535_000 are shared by every accessor through one generic get_persistent / set_persistent pair, so the rent policy is uniform and cannot drift between helpers.
• No .unwrap() / .expect(). Absent keys return Option<T> (every get_* goes through get_persistent, which returns Option<V>), so a saturated store or out-of-order upgrade can never panic the contract. Callers decide how to handle a missing value.
• Typed DataKey enum. Derives contracttype for a stable, host-serialisable layout. All required variants are present (Admin, EscrowCounter, MaxMilestones, Paused, Escrow(u64), Milestone(u64, u32), MilestoneCount(u64), ArbiterRegistry, PlatformFeeBps, Treasury).
• bump_escrow_ttl is defensive. It uses storage.has(...) checks while walking an escrow's milestone keys so a partially-initialised escrow never panics, and extends the escrow row, milestone-count key, and each milestone index.

Verification

• cargo build --release --target wasm32-unknown-unknown → passes, no warnings in storage.rs.
• cargo clippy --all-targets -- -D warnings → clean.
• cargo fmt --all -- --check → clean.
• cargo test → 169 passed, 0 failed (incl. snapshot tests).

---

backend/lib/escrowStateMachine.js

Design decisions

• Pure and isolated. The module never touches the DB, network, or clock. Every transition is a pure function of (currentStatus, nextStatus), so it is trivially testable and is the single source of truth for legality — escrowService delegates all move validation to it and lets it throw.
• Transition table is the only rule. TRANSITIONS encodes exactly the required edges (draft→funded, funded→{in_progress,cancelled,expired}, in_progress→{release_requested,disputed,cancelled,expired}, release_requested→{released,disputed}, disputed→{resolved,cancelled}). Terminal states are simply absent from TRANSITIONS, so any attempt to leave them is rejected by construction.
• transition throws a structured error. { code: 'INVALID_TRANSITION', status: 409, from, to } on an illegal move, and leaves escrow.status untouched on rejection.
• validateHistory enforces three invariants beyond single-step checks:
  • Gaps / illegal steps — every consecutive pair must be a legal TRANSITIONS edge, so a history that skips states to reach disputed (e.g. funded → disputed) is caught.
  • Temporal inversions — a later entry timestamped before an earlier one throws TEMPORAL_INVERSION (409).
  • Unknown states — any unrecognised status throws UNKNOWN_STATE (400).
  • Accepts both Date objects and numeric timestamps.
• 100% branch coverage. The suite exercises every row of TRANSITIONS, every terminal state, representative invalid pairs, and every branch of validateHistory (empty, single, gap, inversion, unknown, numeric timestamps). jest --coverage reports 100% stmts / branch / funcs / lines.

Verification

• tests/unit/escrowStateMachine.test.js → 46 passed, branch coverage 100%.

---

backend/services/stellarService.js

Design decisions

• Every RPC call is span + circuit-breaker wrapped. guarded(spanName, op) runs op inside a withSpan (OpenTelemetry) and a shared CircuitBreaker. The breaker is configured to the spec (failureThreshold: 5, timeout: 30_000 cooldown, shared across all four entry points). When OPEN it rejects immediately with { code: 'STELLAR_RPC_UNAVAILABLE' } — callers surface the outage instead of retrying.
• submitTransaction polls with capped exponential backoff. 1s initial → double each attempt → cap 8s, max 10 attempts (all env-overridable: STELLAR_TX_INITIAL_POLL_MS, STELLAR_TX_POLL_CAP_MS, STELLAR_TX_MAX_ATTEMPTS). An ERROR from sendTransaction returns FAILED immediately; a settled transaction returns SUCCESS; exhausting attempts returns TIMEOUT. Returns { hash, status, errorResultXdr? }.
• getContractEvents fans out into 4096-ledger batches. If latestLedger - startLedger > 4096 the range is split into sequential ceil(range / 4096) batches; each batch queries exactly its [startLedger, endLedger] slice and retries up to 3× with a 500ms exponential base before surfacing the error. A range ≤ 4096 is a single request.
• simulateTransaction dry-runs the XDR and returns { success, cost: { cpuInsns, memBytes } } (zeroed on simulation error).
• getStellarCircuitState() exposes 'CLOSED' | 'OPEN' | 'HALF_OPEN' for health endpoints.
• Testable timing. The sleep primitive is injectable via __setSleep so the backoff schedule can be asserted directly.

Verification

• tests/unit/stellarService.test.js → 14 passed: verifies the backoff schedule, that the circuit opens after 5 failures and rejects with STELLAR_RPC_UNAVAILABLE, and that fan-out issues exactly ceil(range / 4096) requests.

---

backend/services/escrowService.js

Design decisions

• One shape per command handler. Validate preconditions → advance the pure state machine (transition/stepTo) → perform all DB writes + the immutable AdminAuditLog row inside a single withTransaction({ isolationLevel: 'Serializable' }) → emit a domain event (fire-and-forget) after commit.
• Double-spend prevention. releaseMilestone checks amount > remainingBalance before the status check, so an overlapping release surfaces as 422 INSUFFICIENT_BALANCE (not 409), and transitions to Released only when the balance hits exactly 0. resolveDispute enforces clientAmount + freelancerAmount === remainingBalance (422 AMOUNT_MISMATCH).
• Defence-in-depth against double-spend. A per-escrow in-process mutex (withEscrowLock) serialises commands touching the same escrow, guaranteeing the read-modify-write is atomic even if the underlying store's isolation is weaker (e.g. the in-memory test client). This is what makes the concurrent double-spend test resolve to exactly one success + one 422.
• Compensation, not rollback. After the transaction commits, emitEscrowEvent is awaited (fire-and-forget). On emission failure the originating write is deliberately not rolled back; instead a failed_events row is written so a downstream relay can replay it.
• State vocabulary mapping. The canonical lifecycle vocabulary (draft/funded/in_progress/...) is decoupled from the DB EscrowStatus enum via SM_TO_DB / DB_TO_SM, with legacy Active / Completed aliases so older rows still resolve.

Verification

• tests/integration/escrowService.test.js → 9 passed: proves the headline acceptance criterion — two concurrent releaseMilestone calls with overlapping amounts yield exactly one success and one 422. Also covers funding, partial release, dispute/resolve, expire, cancel, and idempotent re-funding (409).

---

backend/services/escrowIndexer.js

Design decisions

• Exactly-once via idempotent upsert. Every event is recorded with prisma.contractEvent.upsert({ where: { eventId }, ... }). Replaying an already-indexed event resolves to the update branch, so a crash-and-restart or a duplicate RPC batch never creates a second row. The unit test processes the same event twice and asserts a single contract_events row.
• Crash-safe cursor. On startup the cursor is loaded from the IndexerState row (id = 1), falling back to INDEXER_START_LEDGER. In the main loop the cursor is advanced and persisted after a batch is fully processed and before the loop moves on — we never advance ahead of what is durably stored, so a restart resumes from the first un-indexed ledger and re-fetches safely (thanks to the upsert above).
• Dead-letter queue. Events that cannot be parsed, or whose handler keeps failing after 3 retries (500ms exponential backoff), are pushed to the Redis list indexer:dlq with the original payload + error + attempt count. The batch continues; one poison event can never stall the pipeline.
• Event → handler dispatch. EscrowCreated → fundEscrow, MilestoneApproved → releaseMilestone, DisputeRaised → raiseDispute, DisputeResolved → resolveDispute, EscrowCancelled → cancelEscrow, LockTimeExpired → expireEscrow. Unknown-but-parseable types are recorded but skipped.
• Graceful shutdown. startIndexer() / stopIndexer() manage the loop; SIGTERM/SIGINT lets the current batch finish, persists the cursor, then exits. We never abort a batch mid-flight.

Verification

• tests/unit/escrowIndexer.test.js → 9 passed: idempotent upsert (single row on double processing), parse-error → DLQ, unknown type skipped, handler retry-then-DLQ with attempts: 3, and cursor fetched from cursor+1 and persisted after each batch (not advanced when no new ledger).

---

Test coverage summary

Layer	Test file	Result
State machine	tests/unit/escrowStateMachine.test.js	46 passed · 100% branch
Stellar RPC	tests/unit/stellarService.test.js	14 passed
Escrow write model	tests/integration/escrowService.test.js	9 passed
Indexer	tests/unit/escrowIndexer.test.js	9 passed
Smart contract	cargo test (escrow_contract)	169 passed, 0 failed

Acceptance criteria checklist

• cargo build --release --target wasm32-unknown-unknown passes, no new storage.rs warnings
• cargo test passes all existing contract tests (169 passed)
• escrowStateMachine.test.js 100% branch coverage; every TRANSITIONS row, terminal state, invalid pair, gap, inversion, and 3-step history covered
• stellarService.test.js verifies backoff sequence, circuit opens after 5 failures, fan-out = ceil(range / 4096) requests
• escrowService.test.js concurrent double-spend → one success, one 422
• escrowIndexer.test.js double event → single row; parse error → DLQ; cursor persists per batch
• cargo fmt / cargo clippy -- -D warnings clean

Note on the broader backend suite

Running the full backend jest suite locally surfaces 12 pre-existing/environmental suite failures (e.g. tests/health.test.js → Cannot find module '../services/emailService.js', webhook.test.js needs a live HTTP endpoint, mediaTranscoder/ipfsService need ffmpeg/network). None of these import the five rebuilt modules, none of the 4 target suites are affected, and they are outside the scope of this issue.