Skip to content

IMPLEMENTATION_ANALYTICS_COMPACTION.md

Latest commit

 

History

History
264 lines (185 loc) · 8.78 KB

IMPLEMENTATION_ANALYTICS_COMPACTION.md

File metadata and controls

264 lines (185 loc) · 8.78 KB
Raw

Analytics Snapshot Compaction Implementation Summary

Issue #440: Compact old analytics snapshots to reduce long-term storage costs

Overview

Implemente| Retention Window | Strategy | | ----------------- | ---------------------- | | < 7 days (recent) | All snapshots (1/hour) | | 7-90 days | Last snapshot per day | | > 90 days | Deleted |

Example (100 days of 1 snapshot/hour):

  • Before: 2,376 snapshots (7 days × 24 + 83 days × 24 + 10 days × 24)
  • After: ~251 snapshots (~89.4% storage reduction)
  • Recent 7 days: 168 snapshots retained (full resolution)
  • 7-90 days: 83 daily snapshots retained (1 per day)
  • 90+ days: Deleted (240 snapshots removed) analytics snapshot compaction system that automatically reduces database storage costs by preserving recent high-frequency data while rolling up older snapshots into daily aggregates.

Implementation Details

1. Database Layer (backend/src/db/analyticsDb.ts)

New Function: dbCompactAnalyticsSnapshots()

  • Purpose: Performs the SQL compaction logic with two-phase deletion
  • Parameters:
    • portfolioId: Target portfolio
    • cutoffDays (default: 90): Delete all snapshots older than this
    • recentDays (default: 7): Keep high-frequency data for this period
  • Returns: CompactionStats with metrics (deletedCount, retainedCount, cutoffTimestamp)
  • Strategy:
    • Phase 1: Delete all snapshots older than cutoffDays
    • Phase 2: For snapshots between cutoffDays and recentDays, keep only the last per day using DISTINCT ON (DATE(timestamp))

2. Service Layer (backend/src/services/analyticsService.ts)

New Methods:

  • compactAnalyticsForPortfolio(): Compact snapshots for a single portfolio
    • Validates cutoffDays >= recentDays
    • Delegates to database function
    • Logs success/failure with metrics
  • compactAllPortfolios(): Compact all portfolios
    • Called by the BullMQ worker
    • Iterates through all portfolios
    • Aggregates results and logs summary

3. Queue & Worker (backend/src/queue/workers/analyticsCompactionWorker.ts)

New Worker: analyticsCompactionWorker.ts

  • Processor: processAnalyticsCompactionJob()

    • Extracts cutoffDays/recentDays from job data
    • Calls analyticsService.compactAllPortfolios()
    • Logs completion with portfolio counts and storage metrics

  • Functions:

    • startAnalyticsCompactionWorker(): Initialize worker singleton
    • stopAnalyticsCompactionWorker(): Graceful shutdown
    • getAnalyticsCompactionWorkerStatus(): Runtime status
    • setAnalyticsCompactionSchedulerRegistered(): Scheduler flag

  • Configuration:

    • Worker name: analytics-compaction
    • Concurrency: 1 (sequential per portfolio)
    • Queue: Created on-demand with default job options

4. Queue Configuration (backend/src/queue/queues.ts)

New Queue: analytics-compaction

  • Added QUEUE_NAMES.ANALYTICS_COMPACTION
  • Added AnalyticsCompactionJobData interface
  • Added getAnalyticsCompactionQueue() getter
  • Updated closeAllQueues() to close compaction queue

5. Scheduler (backend/src/queue/scheduler.ts)

New Schedule:

  • Cron: 0 2 * * 0 (Every Sunday at 02:00 UTC)
  • Job name: scheduled-analytics-compaction
  • Repeatable job ID: repeatable-analytics-compaction
  • Triggers automatically on schedule or can be called manually

Integration:

  • Imports getAnalyticsCompactionQueue and setter function
  • Registers repeatable job in startQueueScheduler()
  • Cleans up on stopQueueScheduler()

6. Monitoring & Readiness (backend/src/monitoring/readiness.ts)

Updates:

  • Added analytics compaction queue health check
  • Added analytics compaction worker status monitoring
  • Updated readiness report to include all queues and workers
  • Readiness endpoint (/ready, /readiness) reflects compaction subsystem status

7. Startup Logging (backend/src/config/startupConfig.ts)

Updates:

  • Added analytics-compaction to startup worker list
  • Updated logs to include new worker in active workers

Test Coverage

Unit Tests: backend/src/test/analyticsCompaction.test.ts

Analytics Service Tests:

  • Calls database function with correct parameters
  • Uses default parameters when not provided
  • Rejects invalid parameter combinations (cutoffDays < recentDays)
  • Handles database errors gracefully
  • Processes multiple portfolios
  • Aggregates statistics correctly
  • Handles empty portfolio lists
  • Fails fast on first portfolio error (with proper error propagation)

Unit Tests: backend/src/test/analyticsCompactionWorker.test.ts

Worker Tests:

  • Calls compactAllPortfolios with default parameters
  • Passes custom parameters through
  • Handles empty results
  • Propagates errors correctly
  • Handles missing correlation IDs

API Paths Impacted

No new API endpoints were added (scope limited as requested). The compaction runs automatically via scheduler.

Existing Endpoints Updated:

  • GET /readiness - Now includes analytics-compaction queue and worker status
  • GET /ready - Now includes analytics-compaction queue and worker status
  • GET /api/v1/system/status - Includes analytics-compaction worker runtime status

Logging & Observability

Log Levels:

  • INFO:

    • Compaction cycle start (portfolio count, cutoffDays, recentDays)
    • Per-portfolio compaction (deletedCount, retainedCount)
    • Cycle completion with aggregated stats

  • ERROR:

    • Compaction failures with portfolio ID and error details
    • Parameter validation failures

Correlation Tracking:

  • All jobs include correlationId for request tracing
  • Logs propagate correlation ID for end-to-end tracing

Error Handling

Failure Cases:

  1. Database connection failure → BullMQ retries with exponential backoff (5s → 80s)
  2. Parameter validation → Rejected immediately with error
  3. Portfolio processing error → Entire batch fails (stops at first error)
  4. Redis unavailable → Queue/worker disabled gracefully

Failure Logs:

  • Include portfolio ID, error message, and correlation ID
  • Severity level appropriate to issue type

Compaction Strategy

Three-Tier Retention:

Age Range Retention Policy
< 7 days (recent) All snapshots (1/hour)
7-90 days Last snapshot per day
> 90 days Deleted

Example:

  • Before: 2,568 snapshots (100 days of hourly data)
  • After: ~180 snapshots (93% storage reduction)
  • Recent 7 days: 168 snapshots retained
  • 7-90 days: ~12 daily snapshots retained
  • 90+ days: Deleted

Files Modified

  1. backend/src/db/analyticsDb.ts - Added compaction DB function
  2. backend/src/services/analyticsService.ts - Added service methods
  3. backend/src/queue/queues.ts - Added queue and job data type
  4. backend/src/queue/workers/analyticsCompactionWorker.ts - NEW
  5. backend/src/queue/scheduler.ts - Added schedule registration
  6. backend/src/monitoring/readiness.ts - Added health checks
  7. backend/src/config/startupConfig.ts - Updated startup logs

Files Created

  1. backend/src/queue/workers/analyticsCompactionWorker.ts - Worker implementation
  2. backend/src/test/analyticsCompaction.test.ts - Service tests
  3. backend/src/test/analyticsCompactionWorker.test.ts - Worker tests
  4. backend/src/db/ANALYTICS_COMPACTION.md - Detailed documentation

Running Compaction

Automatic:

  • Runs every Sunday at 02:00 UTC via BullMQ scheduler

Manual (development):

// Direct service call
const stats = await analyticsService.compactAnalyticsForPortfolio(
  "portfolio-id",
  90, // cutoffDays
  7, // recentDays
);

// All portfolios
const allStats = await analyticsService.compactAllPortfolios(90, 7);

Acceptance Criteria Fulfillment

API/service behavior reachable through intended code path:

  • Compaction triggered via BullMQ worker processing scheduled jobs

  • Database functions executed correctly with SQL logic

  • Service aggregates results across portfolios

    Failure cases return actionable responses/logs:

  • Parameter validation with specific error messages

  • Database failures logged with portfolio ID

  • Correlation ID tracking for tracing

    Automated coverage for touched backend paths:

  • Unit tests for analytics service methods

  • Unit tests for worker processor function

  • Tests cover success paths, error cases, and edge cases

    Changes within scope of issue:

  • Only backend changes (no API/frontend modifications)

  • Focused on analytics snapshot compaction

  • Preserves existing analytics functionality

Future Enhancements (Out of Scope)

  • Admin API endpoint for manual compaction trigger
  • Configurable retention policies per portfolio
  • Archive to cold storage instead of delete
  • Compaction metrics dashboard
  • Rollup statistics (min/max/avg) instead of sampling