Engineering Roadmap: Production-Grade LedgerLens Core

[Inkman007]

Objective

This issue is the coordination point for the LedgerLens engineering backlog overhaul. It replaces file-by-file cleanup placeholders with a finite, evidence-based roadmap. Child issues are independently deliverable; this issue tracks sequencing, cross-cutting decisions, and production-readiness gates.

Operating principles

• Issues must describe an observed gap, bounded scope, measurable acceptance criteria, and validation.
• Production correctness, security, and reproducibility take precedence over feature breadth.
• Changes crossing schemas or repositories require compatibility tests and explicit rollout/rollback plans.
• Experimental ML work must ship behind evaluation and rollout gates.
• No child issue is complete with tests or documentation deferred to an unspecified follow-up.

Workstreams

• Eliminate Suppressed Static-Analysis Debt and Enforce a Zero-Waiver Quality Baseline #607 — Eliminate Suppressed Static-Analysis Debt and Enforce a Zero-Waiver Quality Baseline
• Rebuild the Delivery Pipeline with Verified Artifacts, Environment Promotion, and Automated Rollback #608 — Rebuild the Delivery Pipeline with Verified Artifacts, Environment Promotion, and Automated Rollback
• Restore Automated Versioning, Changelog Generation, and Signed Multi-Artifact Releases #609 — Restore Automated Versioning, Changelog Generation, and Signed Multi-Artifact Releases
• Implement a Production-Faithful Cross-Repository End-to-End Scoring and Contract-Publishing Test #610 — Implement a Production-Faithful Cross-Repository End-to-End Scoring and Contract-Publishing Test
• Learn Calibrated Per-Asset Benford Thresholds with Sparse-Pair Fallbacks and Drift Controls #611 — Learn Calibrated Per-Asset Benford Thresholds with Sparse-Pair Fallbacks and Drift Controls
• Extract Federated Coordination into an Independently Deployable, Protocol-Versioned Service #612 — Extract Federated Coordination into an Independently Deployable, Protocol-Versioned Service
• Build a Reorg-Safe Cross-Chain Correlation Pipeline for Stellar, EVM, and Solana Activity #613 — Build a Reorg-Safe Cross-Chain Correlation Pipeline for Stellar, EVM, and Solana Activity
• Implement the pgvector Embedding Backend with Online Migration and Recall/Latency Parity Gates #614 — Implement the pgvector Embedding Backend with Online Migration and Recall/Latency Parity Gates
• Establish Reproducible Dataset-to-Model Lineage and Risk-Based Promotion Gates #615 — Establish Reproducible Dataset-to-Model Lineage and Risk-Based Promotion Gates
• Prove End-to-End Tenant Isolation Across APIs, Stores, Caches, Metrics, and Background Workers #616 — Prove End-to-End Tenant Isolation Across APIs, Stores, Caches, Metrics, and Background Workers
• Design and Validate Backup, Point-in-Time Recovery, and Disaster-Recovery Procedures #617 — Design and Validate Backup, Point-in-Time Recovery, and Disaster-Recovery Procedures
• Standardize API Failure Semantics, Idempotency, Deadlines, and Backpressure Across REST, GraphQL, gRPC, and WebSockets #618 — Standardize API Failure Semantics, Idempotency, Deadlines, and Backpressure Across REST, GraphQL, gRPC, and WebSockets
• Create a Typed Startup Preflight and Configuration Compatibility Layer for Every Runtime Mode #619 — Create a Typed Startup Preflight and Configuration Compatibility Layer for Every Runtime Mode
• Implement Service-Level Objectives, Burn-Rate Alerting, and Traceable Pipeline Health #620 — Implement Service-Level Objectives, Burn-Rate Alerting, and Traceable Pipeline Health
• Build a Safe Soroban Contract Deployment, Upgrade, and State-Migration Framework #621 — Build a Safe Soroban Contract Deployment, Upgrade, and State-Migration Framework
• Verify Exactly-Once Effects Across Streaming Checkpoints, Retries, Reordering, and Backpressure #622 — Verify Exactly-Once Effects Across Streaming Checkpoints, Retries, Reordering, and Backpressure
• Unify Dependency, Lockfile, and Optional-Feature Management Across Python, Rust, Go, and TypeScript #623 — Unify Dependency, Lockfile, and Optional-Feature Management Across Python, Rust, Go, and TypeScript
• Publish an Executable Architecture Guide and Contributor Decision Map #624 — Publish an Executable Architecture Guide and Contributor Decision Map

Existing advanced backlog retained

• Storage and point-in-time correctness: Add PostgreSQL Backend Option with Read Replicas and Connection Pooling #308, Implement Point-in-Time Correct Feature Store v2 with Time-Travel Queries #310
• Advanced ML and detection: Implement a Temporal Graph Network for Streaming Graph Embedding Updates #313, Add Self-Supervised Contrastive Pretraining for Cold-Start Wallet Embeddings #314, Build an Online Incremental Learning Layer Between Full Model Retrains #315, Implement a Cross-Modal Meta-Ensemble Stacking GNN, LSTM, Tabular, and Benford Signals #316, Export Models to ONNX and Serve Batch GNN Inference via Triton #317, Implement a Fairness and Bias Audit Framework Across Wallet Cohorts #344, Port the Benford and SCC Hot Paths to a PyO3 Rust Extension #346, Build a Flash-Loan Price-Manipulation Detector for AMM Pools #355, Fix the Mathematically-Unreachable Causal-Model Refutation Gate in the /scores/{wallet}/causal-explanation Endpoint #386
• Cryptography, federation, and contract security: Implement Threshold BLS Aggregate Signatures for the Oracle Quorum #319, Build a Secure Aggregation Protocol for Federated Learning Gradient Exchange #320, Add HSM-Backed Signing for the Soroban Service Account Key #321, Add mTLS Between core, api, and webhook-worker Internal Services #325, Implement a GDPR-Style Data-Subject Erasure Workflow for Cached Wallet Metadata #349, Add a Homomorphic-Encryption Aggregation Option to the Federated Learning Server #350, Replace the Fake "(2-of-3)-Threshold" SMPC Gradient Reconstruction With a Real Threshold Secret-Sharing Scheme #387
• Supply chain, operations, and performance: Generate SBOMs and Add Container/Dependency Vulnerability Scanning to CI #323, Implement Automated Dependency Updates with Test-Gated Auto-Merge #324, Build a Load and Soak Testing Harness Simulating Production Ingestion Throughput #333, Implement KEDA-Based Autoscaling for the Webhook Worker and Horizon Streamer #336
• SDK and cross-repository contracts: Build a Rust SDK Client for Off-Chain Verifiers and Exchange Integrations #339, Implement Contract Testing Between core and api for the RiskScore Schema #353
• API security/correctness: Consolidate REST and gRPC Rate Limiting Onto a Shared, Horizontally-Scalable Store to Close a ~20x Per-Key Quota Bypass #382

Sequencing

1. Resolve critical correctness/security foundations and restore trustworthy CI/CD.
2. Establish lineage, isolation, recovery, and end-to-end validation.
3. Complete scaling architecture and advanced detection research behind shadow gates.
4. Promote capabilities only after SLOs and rollback paths are demonstrated.

Exit criteria

• Every open engineering issue uses the professional type/area/priority/difficulty/status taxonomy.
• No generic file-cleanup placeholder remains open.
• Critical issues have owners/milestones before a production release.
• Roadmap and documentation link to live issues rather than stale local backlog files.