Skip to content

SYSTEM-DESIGN.md

Latest commit

 

History

History
76 lines (66 loc) · 3.94 KB

SYSTEM-DESIGN.md

File metadata and controls

76 lines (66 loc) · 3.94 KB
Raw

System Design: FinTech Modernization Platform

1. Problem

Modernize a mission-critical legacy payments/settlement core into a cloud-native, event-driven platform without a big-bang cutover, while preserving correctness, auditability, and availability.

2. Non-functional targets (SLOs)

Concern Target
Availability (ledger write API) 99.99% (≈52 min/year error budget)
Ledger post latency p99 < 250 ms, p50 < 40 ms
Throughput 50,000 TPS sustained peak
Durability Zero acknowledged-then-lost postings (event store is system of record)
Recovery Read model rebuildable from event log; RPO ≈ 0 for the write log
Consistency Strong within an account aggregate; eventual for cross-account read models

3. Capacity math (worked, not hand-waved)

Assume 50,000 TPS peak, average event payload ~500 bytes.

  • Write volume: 50,000 events/s x 500 B ≈ 25 MB/s ≈ 2.1 TB/day of raw event log.
  • Partitioning: target ≤ 5,000 msg/s per Kafka partition => ≥ 10 partitions on ledger.events.v1; provision 20 for headroom and consumer parallelism.
  • Account hot-partitioning: keying by aggregateId (account) preserves per-account ordering. Hot accounts (e.g. a funding/clearing account) are mitigated with sub-keying (accountId#shard) for the projection consumer while keeping the canonical stream ordered.
  • Ledger DB writes: one append + one projection upsert per posting. At 50k TPS that is 100k row ops/s; beyond a single Postgres primary => shard the event store by aggregateId hash, or use a log-structured store (e.g. Cassandra/ScyllaDB) for the append-only log with Postgres for projections.
  • Storage growth: ~2.1 TB/day => tiered retention. Hot (30d) in primary, warm/cold offloaded to the lakehouse (see streaming-lakehouse-platform) which also serves analytics and the async projection rebuild.

4. Component view

flowchart TB
  client[Clients] --> gw[strangler-gateway]
  gw -->|"90% (shrinking)"| legacy[legacy-core]
  gw -->|"10% (growing)"| ledger[ledger-service]
  gw --> settle[settlement-orchestrator]
  legacy -->|outbox CDC| kafka[(Kafka)]
  ledger -->|domain events| kafka
  settle -->|REST| ledger
  kafka --> lake[streaming-lakehouse-platform]
  subgraph ledgerInternals [ledger-service]
    cmd[Command side: event store] --> proj[Projection: account_balance]
  end
Loading

5. Key flows

5.1 Post payment (command)

  1. Idempotency check on paymentId.
  2. Validate invariant (overdraft policy) against read model.
  3. Append PaymentPosted to event store (optimistic concurrency via (aggregate_id, sequence_no)).
  4. Update projection; publish event to Kafka.

5.2 Settlement saga (orchestration)

reserve -> postLedger -> notifyRail -> complete; on rail failure after ledger post => reversePayment (compensation) => FAILED. State persisted at each step (saga_instance) for crash recovery.

5.3 Strangler migration

Weighted routing in the gateway shifts /accounts traffic from legacy to ledger-service. Parallel-run reconciliation compares legacy balances (from CDC) against the new projection before increasing weight.

6. Failure modes and mitigations

Failure Mitigation
Process crash mid-posting Event store append is the commit point; projection is replayable
Dual-write loss (legacy) Transactional outbox (ADR-0005)
Duplicate delivery Idempotency keys + consumer idempotency (at-least-once)
Bad deploy Canary via Argo Rollouts with SLO analysis auto-rollback (deploy/k8s/canary)
Rail outage Saga compensation reverses ledger; settlement marked FAILED, retried later
Read model corruption Rebuild from event log (POST /api/admin/rebuild-projection)

7. Observability

  • Actuator + Micrometer/Prometheus on every service (/actuator/prometheus).
  • Golden signals per service; canary analysis queries error rate and p99 latency directly.
  • Correlation id propagated from command through saga and events for end-to-end tracing.