fix(chaos): exclude intentional teardown from drop metrics and redact JWT secret

Address PR review feedback on the WS portfolio feed load test: (1) mark connections intentionalClose before teardown so the close handler does not count them as unexpected drops/failures; (2) redact the JWT secret from persisted results via sanitizeConfig since results are uploaded as public CI artifacts; (3) stop hardcoding JWT_SECRET in performance-test.yml and generate an ephemeral runtime secret instead; (4) resolve ws/jsonwebtoken through a createRequire fallback because ESM import() ignores NODE_PATH; (5) run the CI load test with auth enabled (CHAOS_WS_AUTH_ENABLED=true and CHAOS_WS_JWT_SECRET=$JWT_SECRET) since /ws/portfolio/:id unconditionally requires a valid JWT.

Author: Gpraiz01

.github/workflows/performance-test.yml

@@ -58,14 +58,17 @@ jobs:
           export NODE_ENV=test
           export PORT=3001
           export DATABASE_URL="sqlite::memory:"
-          export JWT_SECRET="test-secret-key-for-ws-load-testing"
+          # Runtime-generated (never hardcoded/committed): avoids serializing a
+          # secret into the workflow or CI artifacts.
+          export JWT_SECRET="$(openssl rand -hex 32)"
           export STELLAR_NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
           export CORS_ORIGINS="http://localhost:3000,http://localhost:5173"
           export LOG_LEVEL="warn"
           export AUTH_ENABLED="false"
           export ENABLE_STARTUP_SELF_TEST="false"
-          # Resolve ws package from backend's node_modules
-          export NODE_PATH="backend/node_modules"
+          # Resolve ws package from backend's node_modules (used via the createRequire
+          # fallback in the load test, since ESM import() ignores NODE_PATH)
+          export NODE_PATH="$GITHUB_WORKSPACE/backend/node_modules"
 
           # Start backend in background (use working-directory approach for reliable PID capture)
           node backend/dist/index.js &
@@ -91,7 +94,10 @@ jobs:
           CHAOS_WS_BATCH_SIZE="10" \
           CHAOS_WS_BATCH_DELAY_MS="200" \
           CHAOS_WS_DURATION_MS="30000" \
-          CHAOS_WS_AUTH_ENABLED="false" \
+          # /ws/portfolio/:id unconditionally requires a valid JWT, so the load
+          # test must sign tokens with the SAME runtime secret the backend uses.
+          CHAOS_WS_AUTH_ENABLED="true" \
+          CHAOS_WS_JWT_SECRET="$JWT_SECRET" \
           CHAOS_WS_RESULTS_DIR="ws-load-test-results" \
           node scripts/chaos/ws-portfolio-feed-load-test.mjs
           LOAD_TEST_EXIT=$?
@@ -121,7 +127,7 @@ jobs:
           echo "**Test Configuration:**" >> $GITHUB_STEP_SUMMARY
           echo "- Concurrent Connections: 50" >> $GITHUB_STEP_SUMMARY
           echo "- Sustain Duration: 30s" >> $GITHUB_STEP_SUMMARY
-          echo "- Auth Enabled: false" >> $GITHUB_STEP_SUMMARY
+          echo "- Auth Enabled: true (runtime-generated JWT secret)" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
           echo "**Purpose:** Periodically verifies WebSocket portfolio feed capacity and documents connection ceiling for planning." >> $GITHUB_STEP_SUMMARY
 
@@ -167,7 +173,9 @@ jobs:
           export NODE_ENV=test
           export DATABASE_URL="sqlite::memory:"
           export REDIS_URL="redis://localhost:6379"
-          export JWT_SECRET="test-secret-key-for-performance-testing"
+          # Runtime-generated (never hardcoded/committed): avoids serializing a
+          # secret into the workflow or CI artifacts.
+          export JWT_SECRET="$(openssl rand -hex 32)"
           export STELLAR_NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
           export CORS_ORIGINS="http://localhost:3000,http://localhost:5173"
           export LOG_LEVEL="warn"

pr_body.md

@@ -1,79 +1,115 @@
-Closes #995
+Closes #1530
 
 ## Summary
 
-Implements a paginated, filterable **rebalance history endpoint** at `GET /portfolio/:id/rebalance-history` that returns past rebalance outcomes for a given portfolio, including failed rebalances with error reasons.
+Implements a comprehensive chaos/load test script (`scripts/chaos/ws-portfolio-feed-load-test.mjs`) that simulates up to thousands of concurrent WebSocket subscriptions against the `portfolioFeed.ts` endpoint in order to measure connection acceptance latency, message delivery latency, and server resource usage under load. The test identifies and documents the practical concurrent-connection ceiling for current infrastructure sizing, enabling data-driven capacity planning.
 
 ---
 
 ## What was added
 
-### New endpoint: `GET /portfolio/:id/rebalance-history`
+### New Script: `scripts/chaos/ws-portfolio-feed-load-test.mjs`
 
-Returns a paginated list of past rebalances for a portfolio. Each record includes:
+A standalone Node.js ESM script that stress-tests the WebSocket portfolio feed endpoint (`/ws/portfolio/:id`) with configurable concurrency. Phase breakdown:
 
-| Field | Description |
+| Phase | Description |
 |-------|-------------|
-| `timestamp` | ISO-8601 datetime of the rebalance |
-| `trigger` | Raw trigger description |
-| `triggerType` | Normalized: `manual`, `auto`, or `circuit_breaker` |
-| `assetsTrades` | Number of asset trades executed |
-| `totalFeeXlm` | Total gas fee in XLM (null if unavailable) |
-| `totalFeeUsd` | Total gas fee in USD (null if unavailable) |
-| `totalSlippageBps` | Total slippage in basis points (null if unavailable) |
-| `status` | `success`, `partial`, or `failed` |
-| `errorReason` | Error description for failed rebalances (null otherwise) |
-
-### Query Parameters (Filters)
-
-| Param | Type | Description |
-|-------|------|-------------|
-| `from` | ISO-8601 string | Lower-bound timestamp filter (inclusive) |
-| `to` | ISO-8601 string | Upper-bound timestamp filter (inclusive) |
-| `trigger_type` | `manual` \| `auto` \| `circuit_breaker` | Filter by trigger type |
-| `status` | `success` \| `partial` \| `failed` | Filter by rebalance outcome |
-| `page` | integer (default: 1) | Page number |
-| `page_size` | integer (default: 50, max: 500) | Records per page |
-| `sort` | `asc` \| `desc` (default: desc) | Sort order by timestamp |
-
-### Response Shape
+| **Phase 1: Health Check** | Verifies the backend is reachable via `GET /health` before starting |
+| **Phase 2: Ramp Up** | Opens WebSocket connections in configurable batches (supports `batch` and `linear` ramp strategies) |
+| **Phase 3: Sustain** | Maintains all open connections for a configurable duration while collecting message delivery metrics and server resource samples |
+| **Phase 4: Teardown** | Gracefully closes all connections with `1000` close code |
+| **Phase 5: Report** | Generates a comprehensive latency and resource report with capacity planning assessment |
+
+### Metrics Collected
+
+| Metric | Description |
+|--------|-------------|
+| **Connection Acceptance Latency** | Time from `new WebSocket()` to `open` event, with P50/P75/P95/P99/max/mean |
+| **First Message Delivery Latency** | Time from `CONNECTION_ACK` to first `PORTFOLIO_VALUE_UPDATE` |
+| **Message Delivery Latency** | Server timestamp vs. client receipt time for all `PORTFOLIO_VALUE_UPDATE` broadcasts |
+| **Message Throughput** | Total messages received and messages-per-second rate during sustain |
+| **Server Resource Usage** | Polls backend `/metrics` endpoint periodically during sustain phase |
+| **Client Overhead** | Tracks heap usage, RSS, and external memory of the load generator itself |
+| **Connection Drop Rate** | Connections that unexpectedly closed during the sustain phase |
+| **Connection Failure Rate** | Connections that failed to establish or receive `CONNECTION_ACK` |
+
+### Ceiling Assessment
+
+The script automatically assesses the practical connection ceiling based on:
+- Connection failure rate (>10% → degraded)
+- Connection drop rate during sustain (>5% → degraded)
+- P95 connection latency (>5s → degraded)
+- P95 message delivery latency (>2s → degraded)
+
+When no issues are detected, it reports the tested concurrency level as the _minimum_ ceiling and recommends re-running with higher concurrency to find the true limit. When issues are detected, it estimates the practical ceiling using a penalty factor.
+
+### Configuration (Environment Variables)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CHAOS_WS_BACKEND_URL` | `http://localhost:3001` | Backend base URL |
+| `CHAOS_WS_CONCURRENT_CONNECTIONS` | `100` | Total WebSocket connections to simulate |
+| `CHAOS_WS_BATCH_SIZE` | `10` | Connections to open per batch |
+| `CHAOS_WS_BATCH_DELAY_MS` | `200` | Delay between batches |
+| `CHAOS_WS_DURATION_MS` | `60000` | How long to sustain load (ms) |
+| `CHAOS_WS_PORTFOLIO_ID_PREFIX` | `load-test-pf` | Prefix for generated portfolio IDs |
+| `CHAOS_WS_JWT_SECRET` | auto-generated | JWT secret for token generation |
+| `CHAOS_WS_AUTH_ENABLED` | `false` | Whether to require JWT auth tokens |
+| `CHAOS_WS_VERBOSE` | unset | Enable debug-level logging |
+| `CHAOS_WS_TIMEOUT_MS` | `10000` | Individual connection timeout |
+| `CHAOS_WS_RAMP_STRATEGY` | `batch` | `batch` or `linear` ramp strategy |
+
+### Usage
+
+```bash
+# Run with defaults (100 connections, 60s sustain)
+npm run test:chaos:ws-load
+
+# Run with custom concurrency
+CHAOS_WS_CONCURRENT_CONNECTIONS=500 CHAOS_WS_DURATION_MS=120000 \
+  npm run test:chaos:ws-load
+
+# Run with auth enabled (requires backend JWT secret)
+CHAOS_WS_AUTH_ENABLED=true CHAOS_WS_JWT_SECRET=my-secret-key \
+  npm run test:chaos:ws-load
+```
+
+### CI Integration
+
+The script is added to the root `package.json` as `test:chaos:ws-load` alongside the existing `test:chaos` script. It can be integrated into CI for periodic capacity verification:
 
 ```json
-{
-  "data": {
-    "history": [ /* PortfolioRebalanceHistoryItem[] */ ],
-    "pagination": {
-      "page": 1,
-      "pageSize": 50,
-      "total": 127,
-      "totalPages": 3
-    },
-    "filters": {
-      "from": null,
-      "to": null,
-      "trigger_type": null,
-      "status": null
-    }
-  }
-}
+"test:chaos:ws-load": "node scripts/chaos/ws-portfolio-feed-load-test.mjs"
 ```
 
-### Acceptance Criteria
+---
+
+## Acceptance Criteria
+
+- ✅ Load test can simulate a **configurable number** of concurrent WS subscriptions (via `CHAOS_WS_CONCURRENT_CONNECTIONS`)
+- ✅ Test produces a report on **latency** (connection, first-message, delivery with P50/P75/P95/P99) and **resource usage** (server `/metrics` + client overhead) under the simulated load
+- ✅ **Practical connection ceiling** is documented for capacity planning (automatic `assessCeiling()` function)
+- ✅ Script follows existing chaos test patterns (`kill-backend-mid-rebalance.mjs`) with consistent logging, env var configuration, and phased execution
+
+---
+
+## Review Fixes
+
+Resolves the technical issues raised in the PR thread:
 
-- ✅ All rebalance outcomes recorded and returned (success, partial, failed)
-- ✅ Failed rebalances include `errorReason` field with the failure description
-- ✅ Response time monitoring: queries exceeding 200ms are logged as warnings
-- ✅ Paginated with `page`, `page_size`, `total`, `totalPages`
-- ✅ Filterable by `from`, `to`, `trigger_type`, `status`
+- **Intentional disconnects no longer counted as failures.** The teardown phase closes every open connection with close code `1000`. Previously the `close` handler incremented `connectionsDropped` for any connection closed while in the `open` state, so the intentional teardown inflated the drop rate to ~100% and corrupted the ceiling assessment and reported metrics. Connections are now marked `intentionalClose` before teardown and their closes are excluded from the drop/failure counts (`dropRate` reflects only unexpected disconnects during the sustain phase).
+- **JWT secret no longer serialized into CI artifacts.** The results JSON previously persisted the full `CONFIG` object, including the JWT secret, and the results file is uploaded as a public workflow artifact (`ws-load-test-report`). The persisted `configuration` now redacts `jwtSecret` via `sanitizeConfig()`. The workflow also no longer hardcodes plaintext `JWT_SECRET` values; it generates an ephemeral random secret at runtime with `openssl rand -hex 32`.
+- **CI job now authenticates correctly.** `portfolioFeed.ts` unconditionally requires a valid JWT, so the load test job now runs with `CHAOS_WS_AUTH_ENABLED=true` and passes the same runtime secret as `CHAOS_WS_JWT_SECRET`, ensuring the tokens signed by the test validate against the backend.
+- **Rebased onto the base branch** (231 commits behind at the time of rebasing) to restore compatibility and keep the PR mergeable.
 
 ---
 
 ## Files Changed
 
 ### New files
-- `backend/src/test/rebalanceHistory.routes.test.ts` — Unit tests covering: 404 for missing portfolio, default pagination, filter passthrough, failed event error reasons, and totalPages calculation.
+- `scripts/chaos/ws-portfolio-feed-load-test.mjs` — Load test script
 
 ### Modified files
-- `backend/src/api/portfolios.routes.ts` — Added `GET /portfolio/:id/rebalance-history` route
-- `backend/src/api/validation.ts` — Added `portfolioRebalanceHistoryQuerySchema` with Zod validation for all query parameters
-- `backend/src/db/rebalanceHistoryDb.ts` — Added `dbGetPortfolioRebalanceHistory()` — parameterised SQL query with dynamic WHERE clause construction, COUNT for total, and status/trigger mapping
+- `package.json` — Added `"test:chaos:ws-load"` script entry
+- `.github/workflows/performance-test.yml` — CI integration for the load test; no longer hardcodes `JWT_SECRET`
+- `pr_body.md` — PR description

scripts/chaos/ws-portfolio-feed-load-test.mjs

@@ -29,22 +29,39 @@
 import { randomBytes } from 'node:crypto'
 import { setTimeout as sleep } from 'node:timers/promises'
 
-// ─── Dynamic import of ws to avoid issues when module not installed ──────────
-let WebSocketImpl
+// ─── Resolve optional runtime deps (ws, jsonwebtoken) ────────────────────────
+// In CI this script runs from the repo root while `ws`/`jsonwebtoken` live in
+// backend/node_modules. ESM `import()` ignores NODE_PATH, so fall back to
+// createRequire (CommonJS, honors NODE_PATH) and finally to the parent project
+// for local installs.
+import { createRequire } from 'node:module'
+
+const requireFromScript = createRequire(import.meta.url)
+
+let WebSocketImpl = null
 try {
   const wsModule = await import('ws')
   WebSocketImpl = wsModule.default || wsModule.WebSocket || wsModule
 } catch (_) {
-  console.error('[CHAOS-WS] The "ws" package is required. Run: npm install ws')
-  process.exit(1)
+  try {
+    const wsModule = requireFromScript('ws')
+    WebSocketImpl = wsModule.WebSocket || wsModule.default || wsModule
+  } catch (__) {
+    console.error('[CHAOS-WS] The "ws" package is required. Run: npm install ws')
+    process.exit(1)
+  }
 }
 
 // Optional: jwt for token generation when auth is enabled
 let jwtModule = null
 try {
   jwtModule = await import('jsonwebtoken')
 } catch (_) {
-  // jwt is optional; only needed when CHAOS_WS_AUTH_ENABLED=true
+  try {
+    jwtModule = requireFromScript('jsonwebtoken')
+  } catch (__) {
+    // jwt is optional; only needed when CHAOS_WS_AUTH_ENABLED=true
+  }
 }
 
 // ─── Configuration ───────────────────────────────────────────────────────────
@@ -67,6 +84,16 @@ const CONFIG = {
 const httpHost = CONFIG.backendUrl.replace(/^https?:\/\//, '')
 const wsBaseUrl = `ws://${httpHost}`
 
+/**
+ * Configuration copy that is safe to persist/upload: the JWT secret must
+ * never be serialized into results files or CI artifacts.
+ */
+function sanitizeConfig(config) {
+  const safeConfig = { ...config }
+  safeConfig.jwtSecret = '[REDACTED]'
+  return safeConfig
+}
+
 // ─── Logging ─────────────────────────────────────────────────────────────────
 
 function ts() {
@@ -200,7 +227,9 @@ function generateTestToken(portfolioId, userId) {
   if (!jwtModule) {
     throw new Error('jsonwebtoken module not available. Install: npm install jsonwebtoken')
   }
-  return jwtModule.default.sign(
+  // Works for both ESM namespace (jwtModule.default) and CJS exports (jwtModule)
+  const jwtApi = jwtModule.default || jwtModule
+  return jwtApi.sign(
     { sub: userId, type: 'access' },
     CONFIG.jwtSecret,
     { expiresIn: '1h' },
@@ -241,6 +270,8 @@ function connectPortfolioFeed(portfolioId, userId, token) {
       firstMessageTime: null,
       messagesReceived: [],
       state: 'connecting',
+      /** Set to true before an intentional teardown close so it is not counted as a drop/failure */
+      intentionalClose: false,
     }
 
     ws.on('open', () => {
@@ -300,7 +331,11 @@ function connectPortfolioFeed(portfolioId, userId, token) {
 
     ws.on('close', (code, reason) => {
       clearTimeout(connectTimeout)
-      if (record.state === 'connecting') {
+      // Intentional teardown (closeAllConnections) must never count as a drop
+      // or failure — those metrics are for unexpected disconnects only.
+      if (record.intentionalClose) {
+        record.state = 'closed-intentional'
+      } else if (record.state === 'connecting') {
         latencyMetrics.connectionsFailed++
         record.state = 'closed-early'
       } else if (record.state === 'open') {
@@ -346,6 +381,9 @@ function closeAllConnections(records) {
   log.info('Closing ' + openRecords.length + ' open connections...')
 
   for (const record of openRecords) {
+    // Mark as intentional so the 'close' handler does not count this teardown
+    // as an unexpected drop or a connection failure.
+    record.intentionalClose = true
     try {
       record.ws.close(1000, 'Load test complete')
     } catch (_) {
@@ -571,7 +609,7 @@ async function runLoadTest() {
     const resultsPath = path.default.join(dir, 'ws-load-test-results.json')
     const resultsData = {
       timestamp: new Date().toISOString(),
-      configuration: CONFIG,
+      configuration: sanitizeConfig(CONFIG),
       connectionSummary: {
         totalAttempted: CONFIG.concurrentConnections,
         succeeded: latencyMetrics.connectionsSucceeded,