Merge branch 'staging' into feat/wallet-archive-shutdown-redaction

Author: Jambox11

.env.example

@@ -31,6 +31,12 @@ WALLET_ENCRYPTION_KEY=your-secret-encryption-key-min-32-chars
 # Mainnet:  https://horizon.stellar.org
 # ------------------------------------------------------------
 STELLAR_HORIZON_URL=https://horizon-testnet.stellar.org
+STELLAR_NETWORK=TESTNET
+STELLAR_HORIZON_MAX_RETRIES=3
+STELLAR_HORIZON_RETRY_BACKOFF_MS=500
+STELLAR_HORIZON_RETRY_JITTER_MS=250
+BALANCE_STALE_THRESHOLD_MS=300000  # 5 minutes
+# Webhook Configuration
 
 # ------------------------------------------------------------
 # Balance Indexer

README.md

@@ -33,6 +33,8 @@ It handles wallet creation, transaction orchestration, fee sponsorship, and on-c
 
 ## API Endpoints
 
+All routes below are served under the `/v1` prefix (e.g. `GET /v1/health`). See [docs/API-VERSIONING.md](docs/API-VERSIONING.md) for the versioning strategy.
+
 ### Health & Monitoring
 
 #### `GET /health`

docs/API-VERSIONING.md

@@ -0,0 +1,50 @@
+# API Versioning Strategy
+
+This document describes how the Mux backend versions its public HTTP API.
+
+## Current approach: URI path versioning
+
+All routes are served under a global `/v1` prefix, applied once in `src/main.ts`:
+
+```ts
+app.setGlobalPrefix('v1');
+```
+
+Individual controllers (e.g. `@Controller('auth')`, `@Controller('wallets')`)
+declare their resource path only; the version prefix is applied globally so
+every route is automatically namespaced (`/v1/auth/authenticate`,
+`/v1/wallets`, etc.). Requests made without the `/v1` prefix return `404 Not
+Found` — there is no unversioned fallback.
+
+## Why URI versioning
+
+- **Explicit and cache-friendly**: the version is visible in the URL, in logs,
+  and in reverse-proxy/CDN routing rules, without relying on a header that
+  intermediaries may strip.
+- **Simple for consumers**: partners and the frontend hard-code a base URL
+  (e.g. `https://api.mux.dev/v1`) rather than needing to set a custom header
+  on every request.
+- **Matches existing NestJS conventions** in this repo — a single
+  `setGlobalPrefix` call versions every controller without per-route
+  decorators.
+
+## Introducing a breaking change (`/v2`)
+
+When a change is not backwards compatible:
+
+1. Add the new/changed controllers under a `v2` path (NestJS's built-in
+   [URI versioning](https://docs.nestjs.com/techniques/versioning) via
+   `app.enableVersioning({ type: VersioningType.URI })` can be adopted at that
+   point to run `v1` and `v2` controllers side by side).
+2. Keep `/v1` serving the previous behavior until consumers have migrated.
+3. Announce the deprecation window for `/v1` in release notes before removal.
+
+## Non-breaking changes
+
+Additive changes (new endpoints, new optional request/response fields) ship
+directly under the current `/v1` prefix — no new version is required.
+
+## Health and monitoring endpoints
+
+`/v1/health` and `/v1/ready` follow the same prefix as every other route, so
+uptime checks and readiness probes must be configured with the `/v1` path.

docs/migration-recovery-runbook.md

@@ -0,0 +1,296 @@
+# Failed Migration Recovery Runbook
+
+## Overview
+
+This runbook provides procedures for detecting, diagnosing, and recovering from failed database migrations in the Mux Backend API.
+
+## Quick Reference
+
+| Scenario | Steps | Recovery Time |
+|----------|-------|---------------|
+| Migration hangs | Check logs → Kill process → Rollback | 5-10 min |
+| Syntax error | Fix schema → Rollback → Retry | 10-15 min |
+| Constraint violation | Backfill data → Rollback → Retry | 15-30 min |
+| Lock timeout | Kill blocking query → Retry | 5 min |
+
+---
+
+## Detection
+
+### Signs of Migration Failure
+
+1. **Application startup fails** with migration error
+2. **Database logs** show:
+   - `ERROR: relation "table_name" already exists`
+   - `ERROR: column "column_name" does not exist`
+   - `deadlock detected`
+   - `statement timeout`
+3. **Metrics** show stuck migration:
+   - Long-running transaction in `pg_stat_activity`
+   - No progress on migration commit
+
+### Check Migration Status
+
+```bash
+# List applied migrations
+psql -U $DB_USER -d $DB_NAME -c "SELECT * FROM _prisma_migrations ORDER BY finished_at DESC LIMIT 10;"
+
+# Find stuck migrations
+psql -U $DB_USER -d $DB_NAME -c "SELECT * FROM _prisma_migrations WHERE finished_at IS NULL;"
+
+# Check long-running transactions
+psql -U $DB_USER -d $DB_NAME -c "SELECT * FROM pg_stat_activity WHERE state = 'active' AND xact_start < NOW() - INTERVAL '5 minutes';"
+```
+
+---
+
+## Recovery Procedures
+
+### Scenario 1: Syntax Error in Migration
+
+**Symptoms:**
+- `ERROR: syntax error at or near...`
+- Migration marked as started but not finished
+
+**Steps:**
+
+1. **Stop the application**
+   ```bash
+   kubectl scale deployment mux-api --replicas=0
+   ```
+
+2. **Identify the failed migration**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT name FROM _prisma_migrations WHERE finished_at IS NULL;"
+   ```
+
+3. **Rollback (Prisma handles this)**
+   ```bash
+   # Prisma automatically rolls back failed migrations
+   npm run prisma:migrate:resolve -- --rolled-back <migration-name>
+   ```
+
+4. **Fix the migration file**
+   - Edit the migration SQL in `prisma/migrations/<timestamp>_<name>/migration.sql`
+   - Correct syntax errors
+
+5. **Retry migration**
+   ```bash
+   npm run prisma:migrate:deploy
+   ```
+
+6. **Restart application**
+   ```bash
+   kubectl scale deployment mux-api --replicas=3
+   ```
+
+### Scenario 2: Constraint Violation
+
+**Symptoms:**
+- `ERROR: duplicate key value violates unique constraint`
+- `ERROR: insert or update on table violates foreign key constraint`
+
+**Steps:**
+
+1. **Analyze constraint violation**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT * FROM table_name WHERE condition;"
+   ```
+
+2. **Fix conflicting data** (backfill or cleanup)
+   ```sql
+   -- Example: Remove duplicates before adding UNIQUE constraint
+   DELETE FROM table_name WHERE id NOT IN (
+     SELECT MIN(id) FROM table_name GROUP BY unique_col
+   );
+   ```
+
+3. **Rollback migration**
+   ```bash
+   npm run prisma:migrate:resolve -- --rolled-back <migration-name>
+   ```
+
+4. **Retry after data fix**
+   ```bash
+   npm run prisma:migrate:deploy
+   ```
+
+### Scenario 3: Lock Timeout
+
+**Symptoms:**
+- `ERROR: canceling statement due to lock timeout`
+- `statement timeout` in logs
+
+**Steps:**
+
+1. **Identify blocking queries**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT blocked_locks.pid, blocked_locks.relation::regclass, blocking_locks.pid, blocking_locks.relation::regclass FROM pg_locks blocked_locks JOIN pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.granted AND NOT blocked_locks.granted WHERE NOT blocked_locks.granted;"
+   ```
+
+2. **Terminate blocking transaction**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND query LIKE '%your-table-name%' AND state = 'active';"
+   ```
+
+3. **Increase lock_timeout** (temporary)
+   ```sql
+   SET lock_timeout = '30 seconds';
+   ```
+
+4. **Retry migration**
+   ```bash
+   npm run prisma:migrate:deploy
+   ```
+
+### Scenario 4: Hung Migration
+
+**Symptoms:**
+- Migration started hours ago
+- No errors in logs
+- Application waiting on migration
+
+**Steps:**
+
+1. **Check migration status**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT * FROM _prisma_migrations WHERE finished_at IS NULL AND started_at < NOW() - INTERVAL '1 hour';"
+   ```
+
+2. **Identify long-running transaction**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT pid, usename, xact_start, state_change, query FROM pg_stat_activity WHERE xact_start < NOW() - INTERVAL '1 hour';"
+   ```
+
+3. **Terminate stuck transaction**
+   ```bash
+   psql -U $DB_USER -d $DB_NAME -c "SELECT pg_terminate_backend(<pid>);"
+   ```
+
+4. **Mark migration as rolled back**
+   ```bash
+   npm run prisma:migrate:resolve -- --rolled-back <migration-name>
+   ```
+
+5. **Investigate root cause** before retry
+   - Check for missing indexes
+   - Verify disk space
+   - Review lock contention
+
+---
+
+## Verification
+
+### After Any Recovery Attempt
+
+1. **Verify database consistency**
+   ```bash
+   npm run prisma:generate
+   npm run prisma:migrate:status
+   ```
+
+2. **Run integrity checks**
+   ```bash
+   npm run db:integrity-check
+   ```
+
+3. **Test critical flows**
+   ```bash
+   npm run test:integration -- --suite=payments
+   npm run test:integration -- --suite=wallets
+   npm run test:integration -- --suite=recovery
+   ```
+
+4. **Monitor application health**
+   ```bash
+   kubectl logs -f deployment/mux-api -c mux-api | grep -E "ERROR|WARN|migration"
+   ```
+
+---
+
+## Prevention
+
+### Best Practices
+
+1. **Test migrations locally first**
+   ```bash
+   docker-compose up -d postgres
+   npm run prisma:migrate:dev
+   ```
+
+2. **Write idempotent migrations**
+   - Use `IF NOT EXISTS` / `IF EXISTS`
+   - Handle both old and new schema during transition
+
+3. **Add data backfill migrations separately**
+   - Split schema changes and data changes
+   - Allows rollback at schema layer
+
+4. **Monitor lock timeouts**
+   - Set `statement_timeout = 30s` for large ALTER TABLE
+   - Use `ALTER TABLE ... CONCURRENTLY` for indexes on large tables
+
+5. **Use feature flags for compatibility**
+   - Support both old and new column names during migration
+   - Clean up old code after deployment
+
+### Example: Safe Schema Evolution
+
+```sql
+-- Migration 1: Add new column
+ALTER TABLE payments ADD COLUMN assetCode TEXT;
+
+-- Migration 2: Populate data (separate, can be retried safely)
+UPDATE payments SET assetCode = currency WHERE assetCode IS NULL;
+
+-- Migration 3: Add constraints
+ALTER TABLE payments ALTER COLUMN assetCode SET NOT NULL;
+
+-- Migration 4: Deprecate old column (after code updated)
+-- ALTER TABLE payments DROP COLUMN currency_old;
+```
+
+---
+
+## Troubleshooting
+
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `relation already exists` | Migration already applied | Check `_prisma_migrations` table, mark as rolled-back |
+| `column does not exist` | Schema mismatch | Regenerate Prisma client: `npm run prisma:generate` |
+| `deadlock detected` | Concurrent migrations | Ensure migrations run serially, check app replicas |
+| `statement timeout` | Large table operation | Increase timeout or break into smaller batches |
+| `disk space low` | Insufficient storage | Add disk space or clean old transaction logs |
+
+---
+
+## Escalation
+
+**Immediate:**
+- Migration stuck > 30 minutes
+- Multiple rollback failures
+- Production data corruption suspected
+
+**Contact:**
+- On-call DBA: `@dba-oncall` (Slack)
+- Database team: `database-team@mux-labs.com`
+- CTO: For critical data loss scenarios
+
+---
+
+## Audit & Compliance
+
+All failed migrations are tracked via `MigrationRecoveryService`:
+- Logged to application logs
+- Recovery actions recorded in service state
+- Use for post-incident analysis
+
+**Retention:** 30 days in recovery service memory (logs permanent in ELK)
+
+---
+
+## Related Documentation
+
+- [Prisma Migrations Guide](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate)
+- [PostgreSQL Transaction Handling](https://www.postgresql.org/docs/current/runtime-config-client.html)
+- [Mux Backend Architecture](../docs/architecture.md)

prisma/migrations/20260723_add_asset_code_to_payment/migration.sql

@@ -0,0 +1,6 @@
+-- Migration: add assetCode field to Payment
+--
+-- assetCode is an optional field that stores ISO 4217 currency code or custom asset identifier.
+-- Used to validate and track which asset is being transferred in a payment.
+
+ALTER TABLE "Payment" ADD COLUMN "assetCode" TEXT;

prisma/migrations/20260724000000_add_user_default_network/migration.sql

@@ -0,0 +1,2 @@
+-- AlterTable: add defaultNetwork preference to User
+ALTER TABLE "User" ADD COLUMN "defaultNetwork" "WalletNetwork";

prisma/migrations/20260724000000_add_user_last_login_metadata/migration.sql

@@ -0,0 +1,8 @@
+-- Migration: add lastLoginIp and lastLoginUserAgent to User
+--
+-- Captures the IP address and User-Agent seen on the user's most recent
+-- successful authentication, alongside the existing lastLoginAt timestamp.
+-- Both columns are nullable so existing rows require no backfill.
+
+ALTER TABLE "User" ADD COLUMN "lastLoginIp" TEXT;
+ALTER TABLE "User" ADD COLUMN "lastLoginUserAgent" TEXT;

prisma/migrations/20260724010000_add_payment_idempotency_key/migration.sql

@@ -0,0 +1,8 @@
+-- AlterTable
+ALTER TABLE "Payment" ADD COLUMN "idempotencyKey" TEXT;
+
+-- CreateIndex
+CREATE UNIQUE INDEX "Payment_idempotencyKey_key" ON "Payment"("idempotencyKey");
+
+-- CreateIndex
+CREATE INDEX "Payment_idempotencyKey_idx" ON "Payment"("idempotencyKey");