Create RPC provider diversity and failover qualification tests

[Inkman007]

Problem

LedgerLens is a security-sensitive Soroban risk oracle. Create RPC provider diversity and failover qualification tests requires coordinated treatment of deterministic execution, bounded resources, stable interfaces, authorization, storage compatibility, and operational recovery. A narrow happy-path implementation is not sufficient.

Objective

Deliver a production-ready implementation of create rpc provider diversity and failover qualification tests. The change must define its observable contract, preserve existing safety properties, and leave contributors and operators with executable verification—not an undocumented proof of concept.

Primary implementation surface

Start with deployment/release tooling, governance and pause controls, replay tooling, runbooks, and testnet automation. Follow the call and data flow into adjacent modules, but keep unrelated refactors outside this issue.

Required engineering work

• Define measurable readiness criteria, owners, alert thresholds, decision authority, and evidence required before production activation.
• Automate reversible rehearsal on an isolated Stellar network, including partial failure, signer loss, stale data, rollback, and reconciliation.
• Record every operational action with stable identifiers and produce a post-action verification report suitable for audit.
• Record the current behavior and proposed design in the PR, including alternatives rejected and the invariant each material change protects.
• Update public types, events, errors, operator tooling, and documentation together when the behavior crosses those boundaries.
• Use checked arithmetic, deterministic ordering, explicit collection/work bounds, and fail-safe defaults. Read-only paths must not introduce persistent writes.

Acceptance criteria

• The PR contains a concise design section covering trust assumptions, authorization boundaries, state transitions, failure modes, and rollback/recovery.
• The implementation is end-to-end; no placeholder, silently ignored error, unbounded loop, or undocumented operational step remains.
• Existing error discriminants and stored data remain compatible. Any intentional ABI/storage change includes versioning, migration, downgrade behavior, and consumer impact.
• Testnet canary and failure-injection scenarios.
• Backup/restore or rollback drills with integrity reconciliation.
• Operator runbook validation by a reviewer who did not author it.
• Relevant native workspace tests, formatting, strict clippy, locked release WASM build, and repository compatibility checks pass.
• Worst-case CPU, memory, ledger reads/writes, event bytes, and encoded input size are measured or tightly bounded; regressions are reported in the PR.
• Documentation includes configuration, concrete usage, monitoring signals, failure diagnosis, and recovery/rollback.

Adversarial cases to cover

At minimum, test the applicable subset of: unauthorized callers, contract-as-caller authorization, malformed encoding, zero and maximum values, maximum-plus-one rejection, stale/expired state, replay, duplicates, reordered operations, partial execution, signer/configuration rotation, unavailable dependencies, and interrupted retry.

Deliverables

1. Design and invariant notes in the PR.
2. Production implementation in the affected contract/tooling paths.
3. Focused unit, integration, adversarial, and compatibility tests.
4. Resource/size evidence for the worst supported case.
5. Updated interface, security, operations, or integration documentation.

Out of scope

• Unrelated formatting or architectural rewrites.
• Changes to the off-chain machine-learning methodology unless strictly required by this interface.
• Weakening an existing security invariant solely to simplify implementation.

Contributor hand-off

Before coding, identify the exact entry points, storage keys, events, errors, and tests affected in a short PR checklist. If the proposed design changes a public ABI, persistent layout, cryptographic transcript, or privileged authority, request maintainer agreement on that design before implementation.