Skip to content

CODEX_CONTEXT.md

Latest commit

 

History

History
685 lines (454 loc) · 20.7 KB

CODEX_CONTEXT.md

File metadata and controls

685 lines (454 loc) · 20.7 KB
Raw

ApexChainx Backend (apexchainx-be) – Codex Context

Overview

This repository powers the backend API for ApexChainx, an SLA automation and outage settlement platform.

It is responsible for:

  • managing outages and RCA
  • calculating SLA performance
  • exposing analytics and audit data
  • brokering contract-aware payout logic
  • handling authentication, payments, wallet state, jobs, disputes, and webhooks through the API surface

Tech Stack

  • Framework: FastAPI
  • Language: Python (3.11+)
  • Database: PostgreSQL via SQLAlchemy
  • Auth: lightweight in-repo auth store
  • Blockchain: Soroban-aware backend bridge with configurable execution mode
  • Validation: Pydantic v2
  • Async: FastAPI + Celery-oriented modules

Core Domains

1. Outage Management

Responsible for:

  • creating outages
  • updating outage status
  • tracking resolution
  • storing metadata (location, services, subscribers)

Key endpoints:

  • GET /outages
  • POST /outages
  • PUT /outages/{id}

2. SLA System

Core business logic.

Responsible for:

  • calculating MTTR
  • determining SLA compliance
  • triggering penalties or rewards
  • invoking smart contracts

Key endpoints:

  • GET /sla/status/{outage_id}
  • POST /sla/calculate
  • POST /sla/execute-payment

Important:

  • SLA depends on severity thresholds
  • Payment logic is tightly coupled with SLA

3. Payments

Responsible for:

  • exposing payment records tied to SLA outcomes
  • tracking transaction status
  • storing transaction history

Key endpoints:

  • POST /payments/process-sla
  • GET /payments/history

Flow: Outage → SLA Calculation → Smart Contract → Payment → Record stored

4. Wallet Management

Responsible for:

  • creating lightweight wallet records
  • retrieving balances and status
  • linking wallets to users

Key endpoints:

  • POST /wallets/create
  • GET /wallets/{user_id}
  • GET /wallets/{address}/balance

SECURITY CRITICAL:

  • private keys are NEVER returned via API
  • private keys are NEVER logged or exposed
  • only public keys and balance information are accessible
  • wallet operations require proper authentication

5. Analytics

Responsible for:

  • MTTR calculations
  • SLA compliance metrics
  • payment analytics

Key endpoints:

  • GET /api/v1/sla/analytics/dashboard
  • GET /api/v1/sla/analytics/trends
  • GET /api/v1/sla/performance/aggregation

6. Audit Logging

Responsible for:

  • recording all state-changing operations
  • correlating events via X-Correlation-ID
  • immutable append-only log

Key endpoints:

  • GET /api/v1/audit

7. Authentication

Responsible for:

  • login
  • registration
  • JWT issuance

Key endpoints:

  • POST /auth/login
  • POST /auth/register

Architecture

Layered Structure

  • API Layer → routes (FastAPI endpoints)
  • Service Layer → business logic and adapters
  • Repository Layer → SQLAlchemy interaction
  • External Layer → contract bridge, Celery, and webhook integrations

Active vs Dormant Modules

Treat the following as active routed runtime modules:

  • auth
  • audit
  • jobs
  • outages
  • payments
  • sla
  • sla_dispute
  • wallets
  • webhooks

Treat the following as lighter-weight or environment-dependent:

  • auth and wallets are functional but currently backed by in-repo stores
  • jobs and webhooks depend on worker infrastructure for full operational behavior
  • contract execution depends on CONTRACT_EXECUTION_MODE (local or contract)

Treat the following as non-routed or legacy helper paths:

  • app/services/outage_store.py

Important Business Flows

SLA Payment Flow

  1. Outage created
  2. Outage resolved
  3. MTTR calculated
  4. SLA evaluated
  5. Contract adapter or local adapter invoked
  6. Payment record generated
  7. Transaction stored in DB

Constraints & Rules

  • All monetary actions must go through the SLA system
  • Payments must be idempotent
  • Wallet operations must avoid private key exposure
  • SLA must be deterministic and reproducible
  • API responses must follow a consistent structure

Security Constraints & Rules

CRITICAL SECURITY REQUIREMENTS:

  • Private Key Protection: Private keys for Stellar wallets are never exposed via API responses, logs, or documentation
  • Credential Management: All sensitive credentials (database passwords, API keys, JWT secrets) must use environment variables
  • Documentation Safety: Examples must use placeholder values clearly marked as non-production
  • Logging Security: Never log sensitive information including partial keys, tokens, or passwords
  • Environment Separation: Testnet and mainnet credentials must be completely separate
  • Access Control: All financial operations require proper authentication and authorization
  • Audit Trail: All payment and SLA operations must be logged for audit purposes

Documentation Standards:

  • Use [REDACTED] or [EXAMPLE] for sensitive placeholder values
  • Include security warnings for any blockchain or financial operations
  • Show secure patterns (environment variables, secure key management)
  • Distinguish clearly between testnet and mainnet examples

Known Gaps (Areas to Generate Issues)

Codex should focus on generating issues for:

Backend Improvements

  • endpoint validation consistency
  • error handling standardization
  • docs alignment with routed runtime
  • contributor clarity around active vs dormant modules

Coding Standards

  • separate routes from business logic
  • validate all inputs with Pydantic
  • keep business logic out of controllers
  • prefer reusable services and repositories
  • treat the routed runtime as source of truth when docs drift

Issue Generation Rules

When generating issues:

  • scope each issue to ONE domain
  • include:
    • title
    • description
    • acceptance criteria
    • affected modules/files
    • dependencies (Blocked By)
  • prefer small, atomic tasks
  • group issues into:
    • foundational
    • feature
    • integration
    • optimization

Cross-Repo Dependencies

This repo depends on:

  • apexchainx-fe → consumes API
  • apexchainx-contracts → executes SLA logic

Important:

  • any change in SLA logic may affect contracts
  • any API shape change affects frontend

Goal for Codex

Generate a structured backlog of issues that:

  • improves backend reliability
  • ensures correctness of SLA + payments
  • prepares system for production scale
  • maintains clean separation of concerns

SLA Disputes Domain

Responsible for:

  • filing and tracking disputes against SLA outcomes
  • linking disputes to originating SLA records
  • providing audit trail for contested settlements

Key endpoints:

  • POST /api/v1/sla/disputes
  • GET /api/v1/sla/disputes
  • GET /api/v1/sla/disputes/{dispute_id}

Key Terms

Term Definition
MTTR Mean Time to Resolve — the primary SLA compliance metric
SLA Service Level Agreement — defines penalty/reward thresholds
Correlation ID UUID injected per request for cross-system tracing
Contract adapter Soroban bridge activated when CONTRACT_EXECUTION_MODE=contract
Local adapter Default in-process SLA execution path

Rate Limiting

The auth domain includes token family tracking and per-user rate limiting. Rate limit state is stored in the database via migration 0008_auth_rate_limiting. Abuse triggers a short-lived backoff enforced in app/core/rate_limiter.py.

Idempotency

SLA settlement and payment execution are idempotent by design. Duplicate execution attempts for the same outage_id are detected and return the existing record rather than creating a second payment. Payment deduplication is enforced at the database level via migration 0011_payment_deduplication.

Correlation IDs

Every request receives an X-Correlation-ID header injected by app/middleware/correlation.py. The value propagates through audit log entries, SLA records, and payment records so that the full lifecycle of any request can be reconstructed from logs alone.

Middleware Stack

Requests pass through two middleware layers before reaching route handlers:

  1. Correlation middleware (app/middleware/correlation.py) — injects or propagates X-Correlation-ID
  2. Payload size guard (app/middleware/payload_size.py) — rejects bodies exceeding MAX_REQUEST_BODY_SIZE_BYTES

Both are applied globally to all routes.

Repository Layer Conventions

  • Each domain has a dedicated repository class under app/repositories/
  • Repositories accept a SQLAlchemy Session and never call each other directly
  • Business logic belongs in app/services/, not in repositories
  • Route handlers call services; services call repositories

Testing Conventions

  • Unit tests for service logic use mocked repositories
  • Integration tests hit in-memory or test-database sessions
  • Config validation tests use pytest with monkeypatched environment variables
  • All tests live under tests/ and use pytest as the runner

Error Response Shape

All error responses follow a consistent envelope:

{
  "detail": "Human-readable error message",
  "correlation_id": "550e8400-e29b-41d4-a716-446655440000"
}

Validation errors from Pydantic return 422 Unprocessable Entity with a detail array describing each field failure.

Adding a New Domain

To add a new domain to the routed runtime:

  1. Create app/models/{domain}.py — Pydantic request/response schemas
  2. Create app/models/orm/{domain}.py — SQLAlchemy ORM model
  3. Create app/repositories/{domain}_repository.py — DB access
  4. Create app/services/{domain}_service.py — business logic
  5. Create app/api/v1/endpoints/{domain}.py — FastAPI route handlers
  6. Register the router in app/api/v1/router.py
  7. Write an Alembic migration under alembic/versions/
  8. Add tests under tests/

SLA Calculation Detail

MTTR (Mean Time to Resolve) is computed as the difference in minutes between the outage created_at timestamp and its resolved_at timestamp. The computed MTTR is compared against severity-specific thresholds defined in app/services/sla/config.py:

  • If MTTR ≤ threshold → reward outcome
  • If MTTR > threshold → penalty outcome

Both penalty amount and reward amount are configurable per severity tier.

Webhook Delivery Infrastructure

Webhook delivery is handled by app/tasks/webhook_tasks.py as a Celery task. When CELERY_TASK_ALWAYS_EAGER=true, tasks execute synchronously in-process (no Redis required). In production, set CELERY_TASK_ALWAYS_EAGER=false and run a Celery worker alongside the API process.

Analytics Snapshot Backfill

The migration 0012_sla_latest_backfill.py populates the is_latest flag on existing SLA records. This flag allows the analytics layer to efficiently query only the most recent SLA result per outage without a subquery on every request.

Bulk Recompute

When SLA policy thresholds change, existing resolved outages can be recomputed in bulk via POST /api/v1/sla/bulk-recompute. The operation is idempotent — re-running it with the same outage IDs produces the same result. Each recompute emits an audit event.

Policy Version Pinning

SLA results are stamped with the policy version that was active at computation time. This allows historical results to be compared against current policy without ambiguity. Policy version is stored on the sla_results table.

Clock Source Normalization

All timestamps are stored and compared in UTC. The SLA calculator normalises input timestamps to UTC before computing MTTR. Clock skew between the submitting client and the server is not compensated — use NTP-synchronized clients in production.

Deterministic MTTR Boundaries

MTTR boundary computation is deterministic: given the same created_at and resolved_at, the same MTTR and SLA outcome are always produced. This property enables safe recompute and contract parity testing.

Outage Lifecycle States

created → updated → resolved
              ↓
         (SLA computed)
              ↓
     penalty or reward outcome
              ↓
     (optional) payment triggered

An outage can only be resolved once. Resolved outages are immutable — subsequent updates return 409 Conflict.

Payment Deduplication

Payment deduplication is enforced at the database level via a unique constraint on (outage_id, payment_type) added in migration 0011_payment_deduplication. Attempting to execute a second payment for the same outage returns the existing record with 409 Conflict.

Token Family Security

The auth system uses token families to detect refresh token reuse attacks. Each refresh creates a new token family member. Using a previously rotated refresh token invalidates the entire family, forcing re-login. This is implemented in app/repositories/token_family_repository.py.

Session Repository

app/repositories/session_repository.py manages active user sessions. Sessions are linked to token families and expire after the JWT TTL. Logout invalidates the session record immediately, preventing further token refresh even within the TTL window.

Analytics Exporter

app/utils/analytics_exporter.py handles CSV and JSON serialisation of SLA analytics. It reads from the sla_analytics_snapshots table populated by migration 0007_sla_analytics_snapshots. Export operations are read-only and do not mutate any records.

Wallet Registry

app/services/wallet_registry.py manages the mapping between entity IDs and Stellar public keys. In the current release this uses an in-memory store. Persistence via the wallets database table (migration 0010_wallet_persistence) is the planned next step.

Job Cleanup

app/services/job_cleanup.py prunes completed and failed job records older than a configurable retention window. This prevents unbounded growth of the jobs table. The cleanup is triggered as a periodic Celery beat task when a worker is running.

Webhook Signing

app/services/webhook_signing.py provides HMAC-SHA256 signing for outgoing webhook payloads. The signing key is the webhook secret stored per-registration. Signature version metadata is written to the webhook_signature_metadata column added in migration 0013_webhook_secret_metadata.

Metrics Service

app/services/metrics.py exposes internal counters for SLA evaluations, payment executions, webhook delivery attempts, and audit log entries. Metrics are in-process only; no external metrics endpoint is exposed in the current release.

Quick Reference

Task Command
Start API uvicorn app.main:app --reload
Run migrations alembic upgrade head
Check migration state alembic current
Run all tests pytest tests/
Run one test file pytest tests/test_outage_lifecycle.py -v
Start Celery worker celery -A app.tasks.celery_app worker --loglevel=info

Celery Worker Start

To run background tasks in worker mode:

celery -A app.tasks.celery_app worker --loglevel=info

Requires CELERY_BROKER_URL and CELERY_RESULT_BACKEND to be set. Set CELERY_TASK_ALWAYS_EAGER=false to route tasks to the worker instead of executing in-process.

Outage Repository Methods

app/repositories/outage_repository.py exposes:

  • create(db, data) — persist new outage
  • get(db, outage_id) — fetch by ID
  • update(db, outage_id, data) — partial update
  • resolve(db, outage_id, mttr, resolved_at) — mark resolved (atomic)
  • list(db, filters, limit, offset) — paginated list with filter support
  • search(db, query, filters) — full-text and filter search

SLA Repository Methods

app/repositories/sla_repository.py exposes:

  • create(db, data) — persist SLA result
  • get_latest(db, outage_id) — fetch most recent result per outage
  • list(db, filters, limit, offset) — paginated result list
  • bulk_create(db, records) — bulk insert for recompute operations
  • mark_latest(db, sla_id) — update is_latest flag

Payment Repository Methods

app/repositories/payment_repository.py exposes:

  • create(db, data) — persist payment record
  • get(db, payment_id) — fetch by ID
  • get_by_outage(db, outage_id) — fetch payment linked to outage
  • update_status(db, payment_id, status, tx_hash) — update after confirmation
  • list(db, filters, limit, offset) — paginated list with date and status filters

Services vs Repositories: Boundary Rules

Layer Allowed Not Allowed
Route handler Call services, return responses Query DB directly, business logic
Service Business logic, call repositories Import other services' repositories
Repository SQLAlchemy queries only Business logic, HTTP calls
Utility Pure functions, no DB/HTTP Side effects

Pydantic Schema Conventions

  • Request schemas are named {Domain}Create, {Domain}Update
  • Response schemas are named {Domain}Response, {Domain}ListResponse
  • ORM models live in app/models/orm/ and are never exposed directly to route handlers
  • All datetime fields use datetime type (not str) — FastAPI serialises to ISO 8601 automatically

Enum Conventions

Enums used across the API are defined in app/models/enums.py. Always reference the enum class, not raw strings, in service and repository code to benefit from type safety and refactoring support.

Key enums:

  • OutageStatus: open, resolved
  • SLAOutcome: penalty, reward
  • PaymentStatus: pending, confirmed, failed
  • SeverityLevel: low, medium, high, critical

Audit Log Event Types

Key event types emitted by app/services/audit_log.py:

Event Type Trigger
outage.created New outage persisted
outage.resolved Outage resolved with MTTR
sla.computed SLA outcome calculated
sla.recomputed Bulk recompute executed
payment.initiated Stellar payment submitted
payment.confirmed On-chain confirmation received
dispute.filed Dispute created
dispute.resolved Dispute closed
auth.login Successful login
auth.logout Session invalidated
auth.failed Failed login attempt

ORM Model Conventions

ORM models live in app/models/orm/. Each model:

  • extends Base from app/db/base_class.py
  • uses __tablename__ matching the migration table name
  • defines id as UUID primary key
  • includes created_at with server default now()
  • never exposes SQLAlchemy internals to API responses

Database Session Management

Sessions are managed via a FastAPI dependency injected into route handlers:

from app.db.session import get_db
from sqlalchemy.orm import Session
from fastapi import Depends

@router.get("/outages")
def list_outages(db: Session = Depends(get_db)):
    ...

Sessions are committed and closed automatically by the dependency. Do not call db.commit() in repositories — commit in services or use explicit transactions.

Settings Module

app/core/config.py uses Pydantic Settings to load and validate all environment variables at startup. Access settings via the get_settings() function (cached singleton). Never read os.environ directly in application code — always go through get_settings().

Lock Module

app/core/lock.py provides a lightweight advisory lock mechanism used to prevent concurrent SLA recompute operations on the same outage. Uses a database-level advisory lock via PostgreSQL pg_try_advisory_lock. Do not use Python threading primitives for cross-process synchronisation.

Cache Module

app/utils/cache.py provides a simple in-process LRU cache for SLA policy configuration reads. Cache entries expire after a configurable TTL. Invalidate the cache after any policy configuration update to ensure the next SLA computation uses the new values.

Logging

app/utils/logging.py configures structured JSON logging. All log entries include:

  • timestamp (UTC)
  • level
  • message
  • correlation_id (if available in request context)
  • service: always apexchainx-be

Do not use print() in application code. Use the logger from app/utils/logging.py.

Stellar Explorer Utility

app/utils/explorer.py constructs Stellar Expert URLs for transactions and accounts based on the active STELLAR_NETWORK setting. Use this utility when building payment records rather than constructing URLs manually.

Wallet Address Utility

app/utils/wallet_address.py validates and normalises Stellar public key format. Call validate_stellar_address(address) before persisting any wallet address. Returns False for addresses that are not valid G-type Stellar public keys.

Correlation Utility

app/utils/correlation.py provides helpers for reading and setting the correlation ID in the current request context. Use get_correlation_id() inside services when you need to attach the ID to audit events or log entries outside of the request-response cycle.