Skip to content

PR_DESCRIPTION_RUNTIME_METRICS.md

Latest commit

 

History

History
93 lines (68 loc) · 3.57 KB

PR_DESCRIPTION_RUNTIME_METRICS.md

File metadata and controls

93 lines (68 loc) · 3.57 KB
Raw

Commit Message

feat: expose Node.js memory, CPU, and event-loop lag via prom-client

Implement a dedicated runtime metrics collector to expose Node.js-specific 
health indicators including heap usage, external memory, and a custom 
histogram for event-loop lag. Integrates with the application lifecycle 
to start on initialization and gracefully clear intervals on shutdown.

Pull Request Description

🎯 Summary

Expose critical Node.js runtime performance metrics to Prometheus so operators can clearly differentiate garbage collection (GC) pressure from event-loop starvation during load spikes. By measuring heap usage and observing setTimeout deviations, we provide precise observability into the JavaScript runtime's health.

📝 Changes

Core Implementation

src/metrics/runtimeMetrics.ts

  • Created fluxora_nodejs_heap_used_bytes and fluxora_nodejs_heap_total_bytes gauges to monitor GC thrashing.
  • Created fluxora_nodejs_external_bytes gauge for tracking C++ addon/buffer memory.
  • Created fluxora_nodejs_event_loop_lag_seconds histogram with fine-grained buckets (0.005s to 10s) to track synchronous thread-blocking work.
  • Implemented startRuntimeMetrics(intervalMs) using a non-blocking setInterval (with .unref()).
  • Implemented stopRuntimeMetrics() for clean teardown.

src/app.ts

  • Integrated startRuntimeMetrics() inside the createApp initialization flow.
  • Hooked stopRuntimeMetrics() into the graceful shutdown sequence via addShutdownHook().

Testing

tests/metrics/runtimeMetrics.test.ts

  • Added comprehensive tests using vi.useFakeTimers() to validate interval execution without slow test execution.
  • Asserted gauge updates and accurate event-loop lag calculation.
  • Verified graceful interval cleanup on stopRuntimeMetrics().

Bug Fixes Included

tests/webhooks/retry.rateLimit.test.ts

  • Fixed an unrelated TypeScript import and syntax error preventing successful type compilation of the test suite (RateLimitStore missing export issue).

Documentation

docs/observability.md

  • Added ## Runtime Performance Metrics section.
  • Documented all 4 new metrics and their configurations (METRICS_SAMPLE_INTERVAL_MS).
  • Provided SRE Alert Thresholding Strategies (e.g., p99 lag > 1s, heap > 85%).

✨ Key Benefits

  • ✅ Event Loop Visibility - Immediately identifies when synchronous operations block the main thread.
  • ✅ Memory Leak Detection - Fine-grained boundaries over heap constraints.
  • ✅ Graceful Lifecycle - Prevents dangling timers that stall shutdown mechanisms.
  • ✅ High Coverage - Covered fully by new integration tests.

🧪 Testing

Test Coverage

  • ✅ Interval execution accurately logs telemetry.
  • ✅ TypeScript compilation passes completely across the workspace.
  • ✅ Validated that simulated node event loop lag registers correctly in the prometheus histogram.

Verification Commands

# Type check
pnpm run build

# Run runtime metric tests
npx vitest run tests/metrics/runtimeMetrics.test.ts

📊 Impact

Files Changed

  • Added/Modified: 4 files
  • Source Files: 2 files (app.ts, runtimeMetrics.ts)
  • Test Files: 2 files (runtimeMetrics.test.ts, retry.rateLimit.test.ts)
  • Documentation: 1 file (observability.md)

🎯 Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Code is well-commented
  • Documentation updated (observability.md)
  • Tests added for new functionality
  • All tests pass
  • No TypeScript errors
  • Proper graceful shutdown handling implemented