DLQ_HTTP_API_IMPLEMENTATION.md

DLQ HTTP API Implementation Summary

Overview

Successfully implemented HTTP endpoints for DLQ management, enabling operators to trigger replay operations without shell access to pods.

Changes Made

1. New Files Created

Controller & DTOs

• src/api/controllers/dlq.controller.ts

  • POST /admin/dlq/replay - Triggers async replay
  • GET /admin/dlq/status - Returns DLQ metrics
  • Protected by ApiKeyGuard
  • Tracks last replay job metadata in-memory

• src/api/controllers/dto/dlq.dto.ts

  • DlqReplayRequestDto - Request body with optional ids[]
  • DlqReplayResponseDto - 202 response with jobId
  • DlqStatusResponseDto - Status response with depth and replay stats

Tests

• src/api/controllers/dlq.controller.spec.ts

  • Unit tests for controller methods
  • Mocked service dependencies
  • Tests for both replay modes (all vs specific IDs)

• src/test/integration/dlq-api.integration.spec.ts

  • Integration tests against real Postgres database
  • Tests 5+ scenarios:
    • Replay all entries
    • Replay with ledger range filtering
    • Skip non-retryable entries
    • Handle failed replays with retry count increment
    • Dry-run mode verification
  • Uses testcontainers for isolated database

Documentation

• docs/DLQ_API.md
  • Complete API reference
  • Authentication guide
  • Usage patterns (monitoring, CI/CD, K8s CronJob)
  • Security considerations
  • Comparison with CLI tool

2. Modified Files

Module Configuration

• src/api/api.module.ts
  • Added IngestorModule import (provides DlqService)
  • Added DeadLetterEventEntity to TypeORM features
  • Registered DlqController

Architecture

┌─────────────────────────────────────────────────┐
│ HTTP Client (curl, K8s CronJob, CI/CD)         │
└───────────────────┬─────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ DlqController                                   │
│ ├─ POST /admin/dlq/replay → 202 Accepted       │
│ └─ GET /admin/dlq/status → 200 OK              │
│                                                 │
│ Protected by ApiKeyGuard                        │
└───────────────────┬─────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ DlqService                                      │
│ ├─ replayAll(options)                           │
│ ├─ count()                                      │
│ └─ Existing DLQ logic (from CLI)               │
└───────────────────┬─────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ DeadLetterEventEntity (Postgres)                │
└─────────────────────────────────────────────────┘

Acceptance Criteria Status

✅ POST /admin/dlq/replay triggers processing of all failed events

• Endpoint implemented with async execution
• Returns 202 Accepted with job ID
• Supports both "replay all" and "replay specific IDs" modes

✅ GET /admin/dlq/status returns metrics

• Returns { depth: number, lastReplayAt: ISO8601, lastReplayCount: number }
• Depth is real-time count from database
• Last replay metadata tracked in-memory per controller instance

✅ Integration test seeds 5 events and replays via endpoint

• Comprehensive integration test suite with 5+ test cases
• Tests against real Postgres database using testcontainers
• Verifies:
  • Replay of all entries
  • Ledger range filtering
  • Non-retryable entry skipping
  • Failed replay handling
  • Dry-run mode

API Examples

Trigger Replay (All Entries)

curl -X POST http://localhost:3000/admin/dlq/replay \
  -H "x-api-key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

# Response:
{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Replay started for all eligible DLQ entries"
}

Trigger Replay (Specific IDs)

curl -X POST http://localhost:3000/admin/dlq/replay \
  -H "x-api-key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ids": ["uuid1", "uuid2", "uuid3"]}'

# Response:
{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Replay started for 3 specific entries"
}

Check Status

curl http://localhost:3000/admin/dlq/status \
  -H "x-api-key: YOUR_KEY"

# Response:
{
  "depth": 42,
  "lastReplayAt": "2024-01-15T10:30:00.000Z",
  "lastReplayCount": 15
}

Security

• Authentication: API key via x-api-key header (configured via INTERNAL_API_KEY env var)
• Guard: Reuses existing ApiKeyGuard from other admin endpoints
• Logging: All operations logged with job IDs for audit trail

Testing

Run Unit Tests

cd indexer
npm test -- dlq.controller.spec

Run Integration Tests

cd indexer
npm run test:integration -- dlq-api.integration.spec

Deployment Notes

1. Environment Variable: Ensure INTERNAL_API_KEY is set in production
2. Network Access: Configure ingress/firewall rules to restrict admin endpoint access
3. Monitoring: Monitor /admin/dlq/status endpoint for DLQ depth alerts
4. Automation: Consider setting up K8s CronJob for periodic replay (see docs/DLQ_API.md)

Future Enhancements

1. Job Status Tracking: Currently only stores last job; could implement persistent job queue
2. Dry-Run via API: Add query parameter ?dryRun=true to preview replay without execution
3. Webhooks: Notify external systems on replay completion
4. Rate Limiting: Add throttling to prevent replay abuse
5. Granular ID Replay: Optimize ID-based replay to avoid ledger-range approach

Comparison: CLI vs HTTP API

Feature	CLI (dlq:replay)	HTTP API
Shell access	Required	Not required
K8s friendly	❌ Needs kubectl exec	✅ Standard HTTP
Async execution	❌ Blocking	✅ Non-blocking (202)
Automation	Limited	Easy (curl, CI/CD)
Auth	Container access	API key
Status tracking	N/A	/status endpoint

Related Files

• Existing DLQ logic: src/ingestor/dlq.service.ts
• CLI tool: src/cli/dlq-replay.command.ts
• Entity: src/database/entities/dead-letter-event.entity.ts
• API guard: src/api/api-key.guard.ts