feat(observability): add on-call paging receivers and synthetic WebSocket probe

Alertmanager previously named a receiver `pagerduty-critical` but pointed it at
a local webhook, so nothing actually paged. Replace it with real integrations
and make the WebSocket endpoint's availability directly measured rather than
inferred.

On-call paging (#1492):
- Add `oncall-pagerduty` (Events API v2) and `oncall-opsgenie` receivers, with
  credentials read from files mounted at /etc/alertmanager/secrets so keys stay
  out of version control. Route critical alerts to PagerDuty; switching vendors
  is a one-line receiver change.
- Leave warning and info alerts on their existing non-paging receivers.
- Suppress duplicate pages three ways: group the critical route by
  subsystem+service instead of alertname, pin the Opsgenie alias to that same
  incident identity (PagerDuty derives dedup_key from the group key), and add
  inhibit rules so a critical alert silences same-service warnings and
  BackendDown silences the probe alerts it causes. Also relax the critical
  group_interval from 1m to 5m.

Synthetic WebSocket probe (#1494):
- Verify the blackbox handshake properly: assert the response echoes
  `Upgrade: websocket` and returns the Sec-WebSocket-Accept digest derived from
  the probe key, so a bare 101 from a proxy no longer counts as success.
- Give the portfolio-websocket job its own 30s interval and 15s timeout instead
  of inheriting the 15s global scrape interval.
- Label WebSocketHandshakeFailed with `subsystem: api-gateway` so it groups and
  inhibits alongside the other api-gateway alerts, and add WebSocketProbeStalled
  to catch an absent probe — otherwise a dead exporter looks healthy.

Document the escalation policy, dedup mechanisms, and probe behaviour in
docs/OBSERVABILITY.md.

Closes #1492
Closes #1494

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: manoahLinks

deployment/docker-compose.yml

@@ -191,6 +191,7 @@ services:
       - observability
     volumes:
       - ./observability/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - ./observability/alertmanager/oncall-secrets:/etc/alertmanager/secrets:ro
       - alertmanager_data:/alertmanager
     command:
       - --config.file=/etc/alertmanager/alertmanager.yml

deployment/observability/alertmanager/alertmanager.yml

@@ -1,5 +1,7 @@
 global:
   resolve_timeout: 5m
+  pagerduty_url: https://events.pagerduty.com/v2/enqueue
+  opsgenie_api_url: https://api.opsgenie.com/
 
 route:
   receiver: default
@@ -9,27 +11,37 @@ route:
   repeat_interval: 4h
   routes:
     # ----------------------------------------------------
-    # CRITICAL ALERTS: High-priority, aggressive paging
+    # CRITICAL ALERTS: Page the on-call engineer
+    #
+    # Grouping is deliberately coarse here (subsystem + service instead of
+    # alertname) so one underlying incident that trips several rules collapses
+    # into a single notification group. Alertmanager derives the PagerDuty
+    # dedup_key / Opsgenie alias from that group key, so repeated evaluations of
+    # the same incident update the existing page instead of opening a new one.
+    #
+    # Switch to Opsgenie by pointing this route at `oncall-opsgenie`. Only ever
+    # route to one vendor — listing both would page the on-call twice.
     # ----------------------------------------------------
     - match:
         severity: critical
-      receiver: pagerduty-critical
+      receiver: oncall-pagerduty
+      group_by: ['subsystem', 'service']
       group_wait: 10s
-      group_interval: 1m
+      group_interval: 5m
       repeat_interval: 1h
       routes:
         - match:
             subsystem: portfolio-engine
-          receiver: pagerduty-critical
+          receiver: oncall-pagerduty
         - match:
             subsystem: api-gateway
-          receiver: pagerduty-critical
+          receiver: oncall-pagerduty
         - match:
             subsystem: database
-          receiver: pagerduty-critical
+          receiver: oncall-pagerduty
         - match:
             subsystem: system
-          receiver: pagerduty-critical
+          receiver: oncall-pagerduty
 
     # ----------------------------------------------------
     # WARNING ALERTS: Asynchronous logging / Slack warning
@@ -64,15 +76,65 @@ route:
       group_interval: 10m
       repeat_interval: 24h
 
+# ------------------------------------------------------------------
+# Suppression rules: keep one incident to one page.
+# ------------------------------------------------------------------
+inhibit_rules:
+  # A critical alert is already paging the on-call engineer, so the warning and
+  # info alerts describing the same failing service add no new information.
+  - source_matchers:
+      - severity = critical
+    target_matchers:
+      - severity =~ "warning|info"
+    equal: ['subsystem', 'service']
+
+  # The backend process being down is the root cause of every blackbox probe
+  # failure aimed at it. Page for BackendDown only, not for each dependent probe.
+  - source_matchers:
+      - alertname = BackendDown
+    target_matchers:
+      - alertname =~ "BackendReadinessFailed|BackendApiRootFailed|ApiDocsProbeFailed|WebSocketHandshakeFailed"
+
 receivers:
   - name: default
     webhook_configs:
       - url: http://host.docker.internal:5001/alerts
         send_resolved: true
 
-  - name: pagerduty-critical
-    webhook_configs:
-      - url: http://host.docker.internal:5001/alerts/critical
+  # On-call paging via PagerDuty Events API v2. The integration routing key is
+  # read from a mounted secret file so it never lands in version control; see
+  # deployment/observability/alertmanager/oncall-secrets/README.md.
+  - name: oncall-pagerduty
+    pagerduty_configs:
+      - routing_key_file: /etc/alertmanager/secrets/pagerduty_routing_key
+        severity: critical
+        client: stellar-portfolio-rebalancer
+        description: '{{ or .CommonLabels.subsystem "platform" }}/{{ or .CommonLabels.service "unknown" }}: {{ or .CommonAnnotations.summary .CommonLabels.alertname }}'
+        send_resolved: true
+        details:
+          alertname: '{{ .CommonLabels.alertname }}'
+          subsystem: '{{ .CommonLabels.subsystem }}'
+          service: '{{ .CommonLabels.service }}'
+          firing_alerts: '{{ .Alerts.Firing | len }}'
+          description: '{{ .CommonAnnotations.description }}'
+          runbook: 'docs/OBSERVABILITY.md#on-call-escalation-policy'
+
+  # Opsgenie equivalent of the receiver above. `alias` pins the deduplication
+  # identity to the incident (subsystem + service) so Opsgenie keeps appending to
+  # the open alert rather than creating a new one per evaluation cycle.
+  - name: oncall-opsgenie
+    opsgenie_configs:
+      - api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
+        priority: P1
+        alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}'
+        message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})'
+        description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}'
+        source: alertmanager
+        # Opsgenie tags are a single comma-separated string, not a YAML list.
+        tags: 'stellar-portfolio-rebalancer,critical'
+        responders:
+          - name: stellar-portfolio-oncall
+            type: team
         send_resolved: true
 
   - name: slack-warnings

deployment/observability/alertmanager/oncall-secrets/.gitignore

@@ -0,0 +1,2 @@
+pagerduty_routing_key
+opsgenie_api_key

deployment/observability/alertmanager/oncall-secrets/README.md

@@ -0,0 +1,30 @@
+# Alertmanager on-call secrets
+
+This directory is mounted read-only into the Alertmanager container at
+`/etc/alertmanager/secrets`. Alertmanager reads the paging credential from a file
+rather than an inline config value, so the key never enters version control.
+
+Create exactly one file, matching the vendor the `severity: critical` route in
+`../alertmanager.yml` points at:
+
+| Vendor    | File name               | Contents                                            |
+| --------- | ----------------------- | --------------------------------------------------- |
+| PagerDuty | `pagerduty_routing_key` | The Events API v2 integration key for the service   |
+| Opsgenie  | `opsgenie_api_key`      | The API key of an Opsgenie API integration          |
+
+```bash
+printf '%s' "$PAGERDUTY_ROUTING_KEY" > pagerduty_routing_key
+chmod 600 pagerduty_routing_key
+```
+
+Using `printf` rather than `echo` avoids a trailing newline. Do not commit the
+resulting file — the `.gitignore` in this directory already excludes both key
+names.
+
+The directory is named `oncall-secrets` rather than `secrets` because the
+repository root `.gitignore` excludes every `secrets/` directory, which would
+have hidden this README from review as well.
+
+Until a key file exists, critical alerts will fail to deliver and Alertmanager
+will log a notification error. Local development that does not need paging can
+point the critical route at the `default` webhook receiver instead.

deployment/observability/blackbox/blackbox.yml

@@ -7,6 +7,9 @@ modules:
       method: GET
       preferred_ip_protocol: ip4
 
+  # Synthetic WebSocket probe. Sends a real RFC 6455 opening handshake and only
+  # reports success when the server completes the protocol switch, so a plain
+  # HTTP response from a partially-started backend still counts as a failure.
   websocket:
     prober: http
     timeout: 10s
@@ -19,4 +22,9 @@ modules:
         Sec-WebSocket-Version: "13"
         Sec-WebSocket-Key: "x3JJHMbDL1EzLkh9GBhXDw=="
       valid_status_codes: [101]
+      fail_if_header_not_matches:
+        - header: Upgrade
+          regexp: (?i)websocket
+        - header: Sec-WebSocket-Accept
+          regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
       preferred_ip_protocol: ip4

deployment/observability/prometheus/alerts.yml

@@ -60,10 +60,22 @@ groups:
         for: 5m
         labels:
           severity: critical
+          subsystem: api-gateway
           service: websocket
         annotations:
           summary: Backend WebSocket handshake is failing
-          description: The external WebSocket probe is unable to complete a handshake to the backend.
+          description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing.
+
+      - alert: WebSocketProbeStalled
+        expr: absent(probe_success{job="portfolio-websocket", instance="http://backend:3001/"})
+        for: 10m
+        labels:
+          severity: warning
+          subsystem: api-gateway
+          service: websocket
+        annotations:
+          summary: Synthetic WebSocket probe is not reporting
+          description: Prometheus has no probe_success sample for the WebSocket endpoint, so availability is currently unmonitored. Check that the blackbox-exporter container is running and reachable.
 
       - alert: Elevated5xxRate
         expr: sum(rate(stellar_portfolio_http_requests_total{status_code=~"5.."}[5m])) / clamp_min(sum(rate(stellar_portfolio_http_requests_total[5m])), 1) > 0.05

deployment/observability/prometheus/prometheus.yml

@@ -38,7 +38,13 @@ scrape_configs:
       - target_label: __address__
         replacement: blackbox-exporter:9115
 
+  # Synthetic uptime probe for the WebSocket endpoint. The backend upgrades any
+  # non-/ws/portfolio/ path onto the robust broadcast socket, so the root URL is
+  # the externally reachable WS entrypoint. Probed on its own 30s interval rather
+  # than the 15s global default to keep handshake churn on the socket low.
   - job_name: portfolio-websocket
+    scrape_interval: 30s
+    scrape_timeout: 15s
     metrics_path: /probe
     params:
       module: [websocket]

docs/OBSERVABILITY.md

@@ -23,6 +23,31 @@ The current deployment probes:
 
 The blackbox configuration is stored in `deployment/observability/blackbox/blackbox.yml`, and Prometheus scrape jobs are defined in `deployment/observability/prometheus/prometheus.yml`.
 
+### Synthetic WebSocket probe
+
+WebSocket availability is measured directly rather than inferred from HTTP metrics. The `websocket` module in `blackbox.yml` issues a real RFC 6455 opening handshake — `Connection: Upgrade`, `Upgrade: websocket`, `Sec-WebSocket-Version: 13` and a fixed `Sec-WebSocket-Key` — and only records success when the server:
+
+1. answers with status `101 Switching Protocols`,
+2. echoes an `Upgrade: websocket` response header, and
+3. returns the `Sec-WebSocket-Accept` digest derived from the probe's key.
+
+Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line.
+
+The probe target is the backend root URL. `backend/src/index.ts` routes every upgrade request that is not under `/ws/portfolio/` onto the robust broadcast socket, so the root URL is the externally reachable WS entrypoint and needs no authentication to complete a handshake.
+
+The `portfolio-websocket` scrape job runs the probe every 30s with a 15s timeout — its own interval rather than the 15s global default, to keep handshake churn on the socket low while still detecting an outage inside one alert evaluation window.
+
+Failures feed the existing Prometheus/Alertmanager pipeline through two rules in `prometheus/alerts.yml`:
+
+| Alert | Fires when | Severity | Route |
+| --- | --- | --- | --- |
+| `WebSocketHandshakeFailed` | `probe_success == 0` for 5m | critical | pages on-call, suppressed while `BackendDown` is firing |
+| `WebSocketProbeStalled` | no `probe_success` sample for 10m | warning | non-paging warnings channel |
+
+`WebSocketProbeStalled` covers the blind spot where the exporter itself is down: without it, a missing probe looks identical to a healthy one.
+
+To probe a deployed environment, add its public WS origin to the `portfolio-websocket` job targets. Use the `http://` or `https://` scheme (not `ws://`) — the blackbox HTTP prober performs the upgrade over an ordinary HTTP request.
+
 ## Backend
 
 Backend observability is enabled with environment variables in [backend/.env.example](C:\Users\HP\Documents\students\drips\stellar-portfolio-rebalancer\backend.env.example).
@@ -119,6 +144,7 @@ Prometheus alerts are preconfigured for:
 - backend metrics endpoint down
 - backend readiness failures
 - frontend uptime failures
+- WebSocket handshake failures and a stalled WebSocket probe
 - elevated backend 5xx rate
 - failed rebalance queue jobs
 - stale Reflector price rows observed in the last 15 minutes
@@ -132,7 +158,41 @@ The backend exports dedicated price-quality metrics:
 - `stellar_portfolio_reflector_stale_prices_total`
 - `stellar_portfolio_reflector_fallback_usage_total`
 
-Alertmanager ships alerts to `http://host.docker.internal:5001/alerts` by default. Replace that receiver with your Slack, PagerDuty, Opsgenie, or webhook destination before production rollout.
+Alertmanager ships non-critical alerts to `http://host.docker.internal:5001/alerts` by default. Replace those receivers with your Slack or webhook destination before production rollout. Critical alerts page the on-call engineer instead — see below.
+
+## On-Call Escalation Policy
+
+Alert routing lives in `deployment/observability/alertmanager/alertmanager.yml`. Severity decides the channel, and only `critical` wakes a human:
+
+| Severity | Receiver | Channel | Group wait | Re-notify |
+| --- | --- | --- | --- | --- |
+| `critical` | `oncall-pagerduty` | PagerDuty (or Opsgenie) page | 10s | 1h |
+| `warning` | `slack-warnings` | Slack, no paging | 30s | 12h |
+| `info` | `diagnostic-logs` | Log sink, no paging | 1m | 24h |
+
+### Choosing a vendor
+
+Both a PagerDuty and an Opsgenie receiver are defined. The `severity: critical` route points at `oncall-pagerduty`; switch vendors by changing that route's `receiver` to `oncall-opsgenie`. **Route to one vendor only** — pointing at both would page the on-call engineer twice for every incident.
+
+Credentials are read from files mounted read-only at `/etc/alertmanager/secrets`, backed by `deployment/observability/alertmanager/oncall-secrets/` on the host. See the README in that directory for the file names and setup command; the key files themselves are git-ignored. Without a key file, critical alerts fail to deliver and Alertmanager logs a notification error.
+
+### Avoiding duplicate pages
+
+Three mechanisms keep one incident to one page:
+
+1. **Coarse grouping.** The critical route groups by `subsystem` + `service` rather than `alertname`, so several rules tripping on the same outage land in one notification.
+2. **Stable deduplication identity.** Alertmanager derives the PagerDuty `dedup_key` from that group key, and the Opsgenie receiver pins `alias` to `stellar-portfolio-<subsystem>-<service>`. Repeat evaluations update the open incident instead of opening a new one, and the resolved notification closes it.
+3. **Inhibition rules.** A firing `critical` alert suppresses `warning` and `info` alerts for the same `subsystem` + `service`. `BackendDown` additionally suppresses the blackbox probe alerts that depend on the backend (`BackendReadinessFailed`, `BackendApiRootFailed`, `ApiDocsProbeFailed`, `WebSocketHandshakeFailed`), since the process being down is their root cause.
+
+### Escalation path
+
+1. **0–10s** — a critical alert fires; Alertmanager holds it for `group_wait` to collect related alerts into the same page.
+2. **10s** — the page reaches the primary on-call engineer. The payload carries `alertname`, `subsystem`, `service`, the firing-alert count, and a link back to this document.
+3. **Acknowledge and triage.** Cross-check the matching Sentry release and environment tags first — that narrows the search to the exact build that produced the failure. See [TRIAGE.md](TRIAGE.md).
+4. **1h unacknowledged** — Alertmanager re-notifies (`repeat_interval: 1h`). Configure secondary-responder escalation in the PagerDuty/Opsgenie escalation policy itself, not here; Alertmanager only delivers the page.
+5. **Resolution.** Alertmanager sends a resolve notification (`send_resolved: true`) and the incident closes automatically when the underlying alert stops firing.
+
+Adding a new critical alert requires `severity: critical` plus `subsystem` and `service` labels. Without those two labels the alert still pages, but it groups on its own and cannot be inhibited by a related root-cause alert.
 
 ### Queue Operations Dashboard