Skip to content

feat(observability): add on-call paging receivers and synthetic WebSo… #1614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ritik4ever merged 1 commit into from
Jul 30, 2026
Merged

feat(observability): add on-call paging receivers and synthetic WebSo… #1614

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions deployment/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ services:
- observability
volumes:
- ./observability/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
- ./observability/alertmanager/oncall-secrets:/etc/alertmanager/secrets:ro
- alertmanager_data:/alertmanager
command:
- --config.file=/etc/alertmanager/alertmanager.yml
Expand Down
82 changes: 72 additions & 10 deletions deployment/observability/alertmanager/alertmanager.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
global:
resolve_timeout: 5m
pagerduty_url: https://events.pagerduty.com/v2/enqueue
opsgenie_api_url: https://api.opsgenie.com/

route:
receiver: default
Expand All @@ -9,27 +11,37 @@ route:
repeat_interval: 4h
routes:
# ----------------------------------------------------
# CRITICAL ALERTS: High-priority, aggressive paging
# CRITICAL ALERTS: Page the on-call engineer
#
# Grouping is deliberately coarse here (subsystem + service instead of
# alertname) so one underlying incident that trips several rules collapses
# into a single notification group. Alertmanager derives the PagerDuty
# dedup_key / Opsgenie alias from that group key, so repeated evaluations of
# the same incident update the existing page instead of opening a new one.
#
# Switch to Opsgenie by pointing this route at `oncall-opsgenie`. Only ever
# route to one vendor — listing both would page the on-call twice.
# ----------------------------------------------------
- match:
severity: critical
receiver: pagerduty-critical
receiver: oncall-pagerduty
group_by: ['subsystem', 'service']
group_wait: 10s
group_interval: 1m
group_interval: 5m
repeat_interval: 1h
routes:
- match:
subsystem: portfolio-engine
receiver: pagerduty-critical
receiver: oncall-pagerduty
- match:
subsystem: api-gateway
receiver: pagerduty-critical
receiver: oncall-pagerduty
- match:
subsystem: database
receiver: pagerduty-critical
receiver: oncall-pagerduty
- match:
subsystem: system
receiver: pagerduty-critical
receiver: oncall-pagerduty

# ----------------------------------------------------
# WARNING ALERTS: Asynchronous logging / Slack warning
Expand Down Expand Up @@ -64,15 +76,65 @@ route:
group_interval: 10m
repeat_interval: 24h

# ------------------------------------------------------------------
# Suppression rules: keep one incident to one page.
# ------------------------------------------------------------------
inhibit_rules:
# A critical alert is already paging the on-call engineer, so the warning and
# info alerts describing the same failing service add no new information.
- source_matchers:
- severity = critical
target_matchers:
- severity =~ "warning|info"
equal: ['subsystem', 'service']
Comment on lines +85 to +89

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Verify every alert participating in severity-based routing supplies both
# correlation labels before relying on the inhibition rule.
rg -n -C 6 'severity:|subsystem:|service:' deployment/observability/prometheus/alerts.yml

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 15704

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== alertmanager config relevant sections =="
sed -n '1,130p' deployment/observability/alertmanager/alertmanager.yml

echo
echo "== alert rules missing subsystem/service label occurrences =="
python3 - <<'PY'
from pathlib import Path
import re
path=Path('deployment/observability/prometheus/alerts.yml')
text=path.read_text()
# naive rule object by indentation: - alert: starts, labels block may exist
rules=[]
cur=None
for line in text.splitlines():
    if re.match(rb'^\s+-\s+alert:\s+\S+', line):
        cur=''
    if cur is not None:
        cur += line + '\n'
        if re.match(rb'^\s+-\s+alert:\s+\S+', line) and len(cur)>5000:
            cur=None
rules=[]
blocks=[]
depth=None
block=[]
for line in text.splitlines(keepends=True):
    stripped=line.rstrip()
    # simple Prometheus rule-level indent after name interval: alert: is 6 spaces in this file
    if re.match(r'^\s{6}-\s+alert:', line):
        if block:
            blocks.append(''.join(block))
        block=[line]
    elif block:
        indent=len(line) - len(line.lstrip())
        if len(line.strip())==0:
            block.append(line)
            continue
        if re.match(r'^\s{6}-\s+alert:', line):
            # another rule same indentation
            blocks.append(''.join(block))
            block=[line]
        elif re.match(r'^\s{0,5}-\s+', line):
            pass
        else:
            block.append(line)
if block: blocks.append(''.join(block))
missing=[]
for b in blocks:
    m = re.search(r'^\s{6}-\s+alert:\s+(\S+)', b, re.M)
    name = m.group(1) if m else '<unknown>'
    labels = re.search(r'^\s*labels:\s*\n((?:[^\n]+(?:\n|$))*)', b, re.M)
    if not labels:
        missing.append((name, 'no labels block'))
        continue
    txt = labels.group(1)
    if not re.search(r'^\s*subsystem:', txt, re.M):
        if not re.search(r'^\s*service:', txt, re.M):
            missing.append((name, 'no subsystem/service labels'))
        elif re.search(r'^\s*severity:(?:\s+critical|warning|info|debug)?', txt, re.M):
            missing.append((name, 'no subsystem label'))
    elif not re2.search(r'^\s*service:', txt, re.M):
        missing.append((name, 'no service label'))
PY

echo
echo "== receiver templates mentioning label substitution/fallback =="
rg -n 'template=|{{\[.*\]|{{ \(.*\)|subsystem|service|severity' deployment/observability/alertmanager deployment/observability -g '*.tmpl' -g '*.yml' | head -200

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 5622

🌐 Web query:

Alertmanager inhibit rule missing labels treated as equal documentation

💡 Result:

In Alertmanager, inhibition rules treat a missing label and a label with an empty value as semantically equivalent [1][2][3]. Consequently, if all label names listed in the equal field of an inhibition rule are absent from both the source and target alerts, the inhibition rule will consider them to match and will apply [1][2]. This behavior can lead to unintended, global suppression if you are not careful, as alerts missing the specified labels may be inhibited by any firing source alert that also lacks those labels [4]. To avoid this "missing equals missing" behavior and ensure that inhibition only occurs when labels are explicitly present and matching, you can use a regular expression matcher in your source or target matchers to enforce that the label must have a non-empty value [5][6]. For example: inhibit_rules: - source_matchers: - alertname = SourceAlert target_matchers: - alertname = TargetAlert - equal_label =~ ".+" # Ensures the label is present and not empty equal: - equal_label This pattern prevents the inhibition rule from applying if the label is missing, as the requirement for a non-empty value will not be satisfied [5]. Alternatively, simply ensuring that your alerting and recording rules consistently attach all required labels can mitigate this issue [4].

Citations:

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python3 - <<'PY'
import re
from pathlib import Path

text = Path('deployment/observability/prometheus/alerts.yml').read_text()
blocks = re.split(r'(?=^\s{6}-\s+alert:)', text, flags=re.M)
missing = []
for b in blocks:
    m = re.search(r'^\s{6}-\s+alert:\s+(\S+)', b, re.M)
    if not m:
        continue
    name = m.group(1)
    labels = re.search(r'^\s*labels:\s*\n((?:[^\n]+(?:\n|$))*)', b, re.M)
    if not labels:
        missing.append((name, 'no labels block'))
        continue
    txt = labels.group(1)
    has_subsystem = bool(re.search(r'^\s*subsystem:', txt, re.M))
    has_service = bool(re.search(r'^\s*service:', txt, re.M))
    if not has_subsystem or not has_service:
        missing.append((name, f'missing subsystem={has_subsystem}/service={has_service}'))

print('alert rules missing required correlation labels:')
for rule, why in missing:
    print(f'- {rule}: {why}')

# Simulate Alertmanager missing-label inhibition equality for this rule.
# A missing label without an explicit regex matcher is considered equal to
# another missing label with the same name.
examples = [
    ({"severity": "critical"}, {"severity": "warning"}),
    ({"severity": "critical"}, {"severity": "info"}),
    ({"severity": "critical", "subsystem": "api-gateway"}, {"severity": "warning", "service": "backend"}),
    ({"severity": "critical"}, {"severity": "warning", "subsystem": "api-gateway"}),
]
for src, tgt in examples:
    equal = ['subsystem', 'service']
    miss_src = all(k not in src for k in equal)
    miss_tgt = all(k not in tgt for k in equal)
    matched = miss_src and miss_tgt
    print(f'{src} | {tgt} -> inhibit={matched}')
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 1511

Do not let queue labels alone determine inhibition scope.

Several severity-routed rules are missing subsystem (BackendApiRootFailed, ApiDocsProbeFailed, and the queue alerts) while others are missing service. Since Alertmanager treats missing labels as equal, an unlabeled critical queue alert could suppress unrelated warnings/info alerts that also lack both required labels. Require both correlation labels in alert definitions, or add non-empty regex constraints to both source and target matchers plus equal.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/alertmanager/alertmanager.yml` around lines 85 - 89,
Update the severity inhibition rule in Alertmanager so queue or other unlabeled
critical alerts cannot inhibit unrelated warning/info alerts: ensure all
affected alert definitions provide non-empty subsystem and service labels, or
constrain both source_matchers and target_matchers to require non-empty values
for subsystem and service while retaining them in equal.


# The backend process being down is the root cause of every blackbox probe
# failure aimed at it. Page for BackendDown only, not for each dependent probe.
- source_matchers:
- alertname = BackendDown
target_matchers:
- alertname =~ "BackendReadinessFailed|BackendApiRootFailed|ApiDocsProbeFailed|WebSocketHandshakeFailed"

receivers:
- name: default
webhook_configs:
- url: http://host.docker.internal:5001/alerts
send_resolved: true

- name: pagerduty-critical
webhook_configs:
- url: http://host.docker.internal:5001/alerts/critical
# On-call paging via PagerDuty Events API v2. The integration routing key is
# read from a mounted secret file so it never lands in version control; see
# deployment/observability/alertmanager/oncall-secrets/README.md.
- name: oncall-pagerduty
pagerduty_configs:
- routing_key_file: /etc/alertmanager/secrets/pagerduty_routing_key
severity: critical
client: stellar-portfolio-rebalancer
description: '{{ or .CommonLabels.subsystem "platform" }}/{{ or .CommonLabels.service "unknown" }}: {{ or .CommonAnnotations.summary .CommonLabels.alertname }}'
send_resolved: true
details:
alertname: '{{ .CommonLabels.alertname }}'
subsystem: '{{ .CommonLabels.subsystem }}'
service: '{{ .CommonLabels.service }}'
firing_alerts: '{{ .Alerts.Firing | len }}'
description: '{{ .CommonAnnotations.description }}'
runbook: 'docs/OBSERVABILITY.md#on-call-escalation-policy'

# Opsgenie equivalent of the receiver above. `alias` pins the deduplication
# identity to the incident (subsystem + service) so Opsgenie keeps appending to
# the open alert rather than creating a new one per evaluation cycle.
- name: oncall-opsgenie
opsgenie_configs:
- api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
priority: P1
alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}'
message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})'
description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}'
Comment on lines +125 to +131

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

docker run --rm \
  -v "$PWD/deployment/observability/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro" \
  prom/alertmanager:v0.27.0 \
  amtool check-config /etc/alertmanager/alertmanager.yml

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 221

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== alertmanager config excerpt =="
sed -n '100,145p' deployment/observability/alertmanager/alertmanager.yml

echo
echo "== referenced config.go alias handling from prometheus/alertmanager v0.27.0 =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/config/config.go | sed -n '800,950p'

echo
echo "== occurrences of operator.alias in v0.27.0 source =="
curl -fsSL https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1 \
  | python3 - <<'PY'
import json, sys
data=json.load(sys.stdin)
tree=json.loads(data['message'])['tree']
for item in tree:
    if item['type'] == 'blob' and item['path'].endswith(('.go','.yaml','.yml')):
        print(item['path'])
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 9153

🌐 Web query:

prometheus alertmanager OpsgenieConfig alias field config.go v0.27.0

💡 Result:

In Prometheus Alertmanager v0.27.0, there is no configurable alias field available in the OpsGenieConfig definition within config.go [1][2][3]. Alertmanager automatically generates the alias field for Opsgenie notifications to facilitate deduplication [4][5]. This alias is derived internally based on the grouping labels (group_by) configured for the specific route in Alertmanager [5]. Because this process is handled automatically within the notification pipeline, users cannot explicitly set or override this field via the configuration file [1][5][6]. Attempts to customize this behavior, such as preventing cross-team deduplication, have historically been addressed via workarounds rather than direct configuration, including: 1. Using external_labels in Prometheus to ensure that labels used in Alertmanager's group_by are unique per cluster or environment [4]. 2. Configuring a prefix on the Opsgenie side within the Prometheus Integration settings (if supported by your Opsgenie plan), which adds a prefix to the alias field for all incoming alerts for that integration [7]. As of v0.27.0, there have been no changes to OpsGenieConfig to introduce an alias field, and feature requests for such functionality have been noted in the project's issue tracker [4][8][7].

Citations:

🌐 Web query:

alertmanager config validation unknown fields error OpsGenieConfig alias

💡 Result:

The "unknown field" error in Alertmanager configuration, particularly when involving OpsGenie or other receiver configurations, typically stems from one of two primary causes: 1. Using Native Alertmanager Syntax in a Kubernetes AlertmanagerConfig CRD The most common cause for "unknown field" errors when using the Prometheus Operator is providing raw Alertmanager configuration (which uses snake_case, e.g., opsgenie_configs, webhook_configs) inside a Kubernetes AlertmanagerConfig Custom Resource Definition (CRD) [1][2]. The AlertmanagerConfig CRD has its own schema (which uses camelCase, e.g., opsgenieConfigs, webhookConfigs) [3][1][2]. - Solution: Ensure your YAML structure conforms to the AlertmanagerConfig CRD specification rather than the native Alertmanager config file format [1][2]. You can verify the expected fields for your specific version using the command kubectl explain alertmanagerconfig.spec.receivers [1]. 2. Feature Mismatch Between Operator and Alertmanager If you are configuring the native Alertmanager configuration file (often via a Kubernetes Secret or Helm values) and receive an "unknown field" error, your Prometheus Operator or Alertmanager version may be outdated and not yet support the specific field you are trying to use [4][5]. - Solution: Verify that both your Prometheus Operator and Alertmanager image versions support the desired configuration field [4][5]. Features like new receiver types or configuration options (e.g., api_key_file) are added incrementally; if the operator manages the config, ensure it is updated to a version that recognizes the new fields [4][5]. Regarding the specific term "alias" in your query: "alias" is not a standard field within the native Alertmanager opsgenie_config schema [6]. If you are encountering an error mentioning "alias" as an unknown field, it is likely an unsupported or incorrectly placed configuration key. If you are attempting to alias an OpsGenie configuration, verify the current Alertmanager documentation to see if that functionality is supported or if it must be handled differently (e.g., using different receiver names) [6].

Citations:

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/alertmanager.yml.example \
  | sed -n '1,180p'

echo
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/config/notifiers.go \
  | sed -n '480,580p'

echo
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/notify/opsgenie/opsgenie.go \
  | sed -n '1,180p'

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 228

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== candidate config/notifier files in v0.27.0 =="
python3 - <<'PY'
import json, urllib.request
url='https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1'
try:
    with urllib.request.urlopen(url) as r:
        data=json.load(r)
except Exception as e:
    print('GH API payload error:', repr(e))
    raise SystemExit(0)
for item in (data.get('tree') or []):
    if item.get('type')=='blob' and item.get('path') in {'config/notifiers.go','notify/opsgenie/opsgenie.go','config/config.go'}:
        print(item['path'], item.get('sha'))
PY

echo
curl -fsSL https://github.com/prometheus/alertmanager/raw/v0.27.0/config/notifiers.go | sed -n '1,320p' | sed -n '440,560p'

echo
curl -fsSL https://github.com/prometheus/alertmanager/raw/v0.27.0/notify/opsgenie/opsgenie.go | sed -n '80,220p'

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 4238

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python3 - <<'PY'
import ssl, urllib.request
ctx=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname=False
ctx.verify_mode=ssl.CERT_NONE
data=json.load(urllib.request.urlopen('https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1', context=ctx))
for item in data.get('tree', []):
    if item.get('path') in {'config/notifiers.go','config/config.go'}:
        print('FETCHING', item.get('path'), item.get('sha'))
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 288

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python3 - <<'PY'
import json, ssl, urllib.request
ctx=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname=False
ctx.verify_mode=ssl.CERT_NONE
data=json.load(urllib.request.urlopen('https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1', context=ctx))
for item in data.get('tree', []):
    if item.get('path') in {'config/notifiers.go','config/config.go'}:
        print('FETCHING', item.get('path'), item.get('sha'))
PY

echo
curl -fsSL --insecure https://github.com/prometheus/alertmanager/raw/v0.27.0/config/notifiers.go | sed -n '420,540p'

echo
curl -fsSL --insecure https://github.com/prometheus/alertmanager/raw/v0.27.0/config/config.go | rg -n "OpsGenieConfig|alias|yaml:" -C 2 | sed -n '1,220p'

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 973

Remove the unsupported Opsgenie alias field.

opsgenie_configs.alias is not part of Alertmanager v0.27.0’s Opsgenie schema, and that notifier sets Opsgenie alias to the Alertmanager group key (key.Hash()), so this unknown field prevents config load.

Proposed fix 
-  # Opsgenie equivalent of the receiver above. `alias` pins the deduplication
-  # identity to the incident (subsystem + service) so Opsgenie keeps appending to
-  # the open alert rather than creating a new one per evaluation cycle.
+  # Opsgenie uses Alertmanager's group key for its alias, preserving
+  # deduplication for this route's subsystem/service grouping.
   - name: oncall-opsgenie
     opsgenie_configs:
       - api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
         priority: P1
-        alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}'
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: oncall-opsgenie
opsgenie_configs:
- api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
priority: P1
alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}'
message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})'
description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}'
- name: oncall-opsgenie
opsgenie_configs:
- api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
priority: P1
message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})'
description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}'
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/alertmanager/alertmanager.yml` around lines 125 -
131, Remove the unsupported alias entry from the oncall-opsgenie
opsgenie_configs configuration, leaving the api_key_file, priority, message, and
description settings unchanged.

source: alertmanager
# Opsgenie tags are a single comma-separated string, not a YAML list.
tags: 'stellar-portfolio-rebalancer,critical'
responders:
- name: stellar-portfolio-oncall
type: team
send_resolved: true

- name: slack-warnings
Expand Down
2 changes: 2 additions & 0 deletions deployment/observability/alertmanager/oncall-secrets/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
pagerduty_routing_key
opsgenie_api_key
30 changes: 30 additions & 0 deletions deployment/observability/alertmanager/oncall-secrets/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Alertmanager on-call secrets

This directory is mounted read-only into the Alertmanager container at
`/etc/alertmanager/secrets`. Alertmanager reads the paging credential from a file
rather than an inline config value, so the key never enters version control.

Create exactly one file, matching the vendor the `severity: critical` route in
`../alertmanager.yml` points at:

| Vendor | File name | Contents |
| --------- | ----------------------- | --------------------------------------------------- |
| PagerDuty | `pagerduty_routing_key` | The Events API v2 integration key for the service |
| Opsgenie | `opsgenie_api_key` | The API key of an Opsgenie API integration |

```bash
printf '%s' "$PAGERDUTY_ROUTING_KEY" > pagerduty_routing_key
chmod 600 pagerduty_routing_key
Comment on lines +15 to +17

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

docker image inspect prom/alertmanager:v0.27.0 --format '{{.Config.User}}'
stat -c '%u:%g %a %n' \
  deployment/observability/alertmanager/oncall-secrets/pagerduty_routing_key

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 221

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== files =="
git ls-files | rg 'deployment/observability/alertmanager|oncall-secrets|README|compose|docker|Dockerfile' || true

echo
echo "== target README =="
if [ -f deployment/observability/alertmanager/oncall-secrets/README.md ]; then
  cat -n deployment/observability/alertmanager/oncall-secrets/README.md
fi

echo
echo "== alertmanager compose/service references =="
rg -n "alertmanager|PAGERDUTY_ROUTING_KEY|pagerduty_routing_key|security_context|users|uid|gid|user:" -S deployment .github 2>/dev/null || true

echo
echo "== upstream alertmanager v0.27.0 Dockerfile relevant lines =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | sed -n '1,220p' | nl -ba

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 8307

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== compose alertmanager block =="
sed -n '180,210p' deployment/docker-compose.yml | cat -v

echo
echo "== alertmanager config relevant routing/receiver =="
sed -n '90,135p' deployment/observability/alertmanager/alertmanager.yml

echo
echo "== upstream v0.27.0 Dockerfile lines =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | awk 'NR<=220{printf "%4d  %s\n", NR, $0}'

echo
echo "== parse dockerfile user directives =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | awk 'toupper($1)=="USER"{users=users (users?"\\n":"") NR": "$0} END{print (users?users:"NO USER DIRECTIVE FOUNDED")}'

echo
echo "== static permissions probe for current README command behavior =="
tmp="$(mktemp -d)"
cd "$tmp"
USER="$(id -u):$(id -g)"
PAGERDUTY_ROUTING_KEY=fake-key
printf '%s' "$PAGERDUTY_ROUTING_KEY" > pagerduty_routing_key
chmod 600 pagerduty_routing_key
stat -c 'mode=%a owner=%u:%g file=%n' pagerduty_routing_key
if [ "$(stat -c '%u' pagerduty_routing_key)" = "$(id -u)" ]; then
  echo "current_host_user_matches_file_owner=true"
else
  echo "current_host_user_matches_file_owner=false"
fi

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 4437

🌐 Web query:

Docker USER nobody UID GID nobody container filesystem permissions mode 0600 readable by nobody

💡 Result:

In a Docker container, a file with 0600 (read/write only for the owner) permissions is readable by the nobody user if and only if that user is the actual owner of the file [1][2]. The nobody user (typically UID 65534) is not a special "superuser" that bypasses standard Unix filesystem permissions; it is simply a restricted, low-privilege account [3][4][5]. When you encounter a situation where a container running as nobody cannot read a 0600 file, it is due to one of the following factors: 1. Ownership Mismatch: If the file is owned by a different UID (even if the usernames match, as UID/GID numbers are what the Linux kernel enforces), the nobody user will be denied access [1][6]. 2. User Namespaces (User Remapping): If you are using rootless Docker or user namespace remapping (userns-remap), the UIDs inside the container are shifted relative to the host [7][6]. A nobody user (UID 65534) inside the container is mapped to a different, high-range UID on the host [4]. If the file on the host is owned by the host's actual nobody user, the container's nobody user will not have access because the kernel sees them as different identities [7][6]. 3. Filesystem Limitations: Docker bind mounts do not automatically translate or map permissions between the host and container [8][1]. The container process must have the appropriate UID/GID match or sufficient permissions (e.g., group or world read access) to interact with the file [9][2]. To resolve access issues for 0600 files: - Ensure the effective UID/GID of the container process matches the UID/GID of the file owner on the host [9][1]. - If you cannot change the process UID, adjust the file permissions on the host to be group or world-readable (e.g., 0640 or 0644), provided this aligns with your security requirements [10][2]. - When using user namespaces, ensure the host-side file ownership corresponds to the remapped UID that the container's nobody user resolves to on the host [6].

Citations:

Make the bind-mounted secret readable by Alertmanager.

prom/alertmanager:v0.27.0 runs as the container nobody UID, while this command creates pagerduty_routing_key owned by the host user with 0600. Alertmanager then cannot read routing_key_file: /etc/alertmanager/secrets/pagerduty_routing_key, and critical PagerDuty notifications fail. Document setup using UID 65534 ownership/a compatible group, or set container permissions that expose the secret safely.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/alertmanager/oncall-secrets/README.md` around lines
15 - 17, Update the secret setup instructions in the README so the bind-mounted
pagerduty_routing_key is readable by Alertmanager’s container user (UID 65534)
while remaining safely restricted. Document applying UID 65534 ownership or an
equivalent compatible group/permissions after creating the file, and keep the
routing_key_file path unchanged.

```

Using `printf` rather than `echo` avoids a trailing newline. Do not commit the
resulting file — the `.gitignore` in this directory already excludes both key
names.

The directory is named `oncall-secrets` rather than `secrets` because the
repository root `.gitignore` excludes every `secrets/` directory, which would
have hidden this README from review as well.

Until a key file exists, critical alerts will fail to deliver and Alertmanager
will log a notification error. Local development that does not need paging can
point the critical route at the `default` webhook receiver instead.
8 changes: 8 additions & 0 deletions deployment/observability/blackbox/blackbox.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ modules:
method: GET
preferred_ip_protocol: ip4

# Synthetic WebSocket probe. Sends a real RFC 6455 opening handshake and only
# reports success when the server completes the protocol switch, so a plain
# HTTP response from a partially-started backend still counts as a failure.
Comment on lines +10 to +12

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files | rg '(^deployment/observability/blackbox/blackbox.yml$|^docs/OBSERVABILITY.md$|^deploy|^blackbox|observability|package|requirements|Makefile|README)'

echo
echo "== blackbox config =="
if [ -f deployment/observability/blackbox/blackbox.yml ]; then
  cat -n deployment/observability/blackbox/blackbox.yml
fi

echo
echo "== observability docs relevant =="
if [ -f docs/OBSERVABILITY.md ]; then
  nl -ba docs/OBSERVABILITY.md | sed -n '1,90p'
fi

echo
echo "== exporter version / pin refs =="
rg -n "blackbox|blackbox_exporter|portworx|websocket|portfolio-websocket|prober:" . \
  --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || true

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 3695

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== observability docs relevant =="
sed -n '1,90p' docs/OBSERVABILITY.md

echo
echo "== exporter / scrape references =="
sed -n '1,220p' deployment/observability/prometheus/prometheus.yml
echo
sed -n '1,160p' .github/workflows/observability-lint.yml
echo
rg -n "blackbox|blackbox_exporter|websocket|portfolio-websocket|prober:" deployment deploy README*.md docs .github --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || true

echo
echo "== scripts observability references =="
sed -n '1,220p' scripts/bootstrap-observability.sh 2>/dev/null || true

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 22393

🌐 Web query:

Prometheus Blackbox Exporter websocket prober PROTOCOL error HTTP responses 101 Connection upgrade documentation

💡 Result:

As of July 30, 2026, Prometheus Blackbox Exporter supports WebSocket probing directly through a dedicated websocket prober [1][2]. If you are encountering issues with HTTP 101 (Switching Protocols) responses during probing, note the following: 1. Dedicated WebSocket Prober: The websocket prober was specifically introduced to handle the WebSocket handshake and connection upgrade process [1]. Ensure you are using prober: websocket in your configuration file rather than the standard http prober, as the http prober is not designed to maintain or interact with an upgraded WebSocket connection [2][3]. 2. Configuration: The websocket prober configuration allows you to define a query_response sequence [2]. This includes an expect field (which uses regular expressions to match incoming server messages) and a send field (to send messages after the connection is successfully upgraded) [2]. 3. Troubleshooting 101 Responses: * The HTTP 101 Switching Protocols status code is the expected successful response for a WebSocket upgrade request [4]. * If your probe is failing, use the debug=true parameter in your probe URL (e.g., /probe?target=...&module=...&debug=true) to view detailed logs of the request and response lifecycle [5]. This will confirm if the server is correctly returning the 101 status and if the handshake is completing successfully [5]. * Verify that your server implementation strictly adheres to the WebSocket handshake requirements defined in RFC 6455, as the client will drop the connection if the protocol switch is not handled correctly [6][4]. For detailed configuration syntax, refer to the official Blackbox Exporter documentation [2]. Ensure your version of Blackbox Exporter is up to date, as this feature was finalized and merged into the main codebase in early 2026 [1].

Citations:

🌐 Web query:

Prometheus Blackbox Exporter v0.25.0 websocket prober configuration reference

💡 Result:

The Prometheus Blackbox Exporter version 0.25.0, released on April 9, 2024, does not include support for a websocket prober [1][2]. WebSocket support was added to the Blackbox Exporter project after the release of v0.25.0 [3]. The feature was introduced via pull request #1278 [3]. If you require WebSocket probing capabilities, you must use a more recent version of the Blackbox Exporter that includes this feature [4][3]. When using a version that supports the websocket prober (e.g., current main branch), the configuration structure is as follows: websocket_probe: # Optional HTTP request configuration http_config: [ <http_client_config> ] # The HTTP headers set for the probe headers: [ : [ values: [,...] ], [ secrets: [,...] ], [ files: [,...] ] ],... # The query sent after connection upgrade and the expected associated response query_response: [ - [ expect: ], [ send: ], [ starttls: <boolean | default = false> ] ],... # The IP protocol of the Websocket probe (ip4, ip6) [ preferred_ip_protocol: | default = "ip6" ] # Fallback to the other IP protocol if the preferred_ip_protocol fails [ ip_protocol_fallback: | default = true ] For examples of how to define this in your configuration file, refer to the official example.yml found in the project repository [5].

Citations:

Use Blackbox Exporter websocket_probe, not an HTTP upgrade probe.

prom/blackbox-exporter:v0.25.0 does not support a prober: websocket, so this http module only performs a one-shot 101 response check and falls back to draining until closed/timeout. Either update Blackbox to a version with websocket_probe support and change:

  • deployment/observability/blackbox/blackbox.yml#L13-L29 to prober: websocket / websocket_probe: configuration, and
  • docs/OBSERVABILITY.md to stop saying the HTTP prober completes the WebSocket handshake.
📍 Affects 2 files
  • deployment/observability/blackbox/blackbox.yml#L10-L12 (this comment)
  • docs/OBSERVABILITY.md#L28-L32
  • docs/OBSERVABILITY.md#L49-L49
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/blackbox/blackbox.yml` around lines 10 - 12, Update
deployment/observability/blackbox/blackbox.yml lines 13-29 to use a Blackbox
Exporter version supporting websocket_probe, with prober: websocket and
websocket_probe configuration instead of the HTTP upgrade probe. Update
docs/OBSERVABILITY.md lines 28-32 and 49 to describe the WebSocket prober and
remove claims that the HTTP prober completes the handshake.

Source: MCP tools

websocket:
prober: http
timeout: 10s
Expand All @@ -19,4 +22,9 @@ modules:
Sec-WebSocket-Version: "13"
Sec-WebSocket-Key: "x3JJHMbDL1EzLkh9GBhXDw=="
valid_status_codes: [101]
fail_if_header_not_matches:
- header: Upgrade
regexp: (?i)websocket
- header: Sec-WebSocket-Accept
regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Comment on lines +25 to +29

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python - <<'PY'
import re

assert re.search(r'(?i)websocket', 'not-websocket')
assert re.fullmatch(r'(?i)websocket', 'not-websocket') is None
assert re.fullmatch(r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=', 'HSmrc0sMlYUkAGmm5OPpG2HaGWk=')
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 177

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '--- blackbox config ---\n'
cat -n deployment/observability/blackbox/blackbox.yml

printf '\n--- search blackbox config files ---\n'
git ls-files | rg '(^|/)blackbox\.y(?:ml|a)ml$' || true

printf '\n--- Prometheus/blackbox_exporter behavior probe from source/docs if present in repo or curl upstream raw ---\n'
python3 - <<'PY'
import re
import urllib.request

patterns = {
    "unanchored_upgrade": r'(?i)websocket',
    "unanchored_accept": r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=',
}
cases = {
    "false_positives": ["not-websocket", "websocket-possible", "abcHSmrc0sMlYUkAGmm5OPpG2HaGWk=xyz", "HSmrc0sMlYUkAGmm5OPpG2HaGWk=extra"],
    "expected_true": ["websocket", "HSmrc0sMlYUkAGmm5OPpG2HaGWk="],
}
for name, pat in patterns.items():
    print(f"\n{name}: {pat}")
    for label, vals in cases.items():
        hits = [v for v in vals if re.search(pat, v)]
        print(f"  {label}: {hits}")

url = "https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/prober/http.go"
try:
    data = urllib.request.urlopen(url, timeout=15).read().decode()
    for needle in ["fail_if_header_not_matches", "MatchString"]:
        idx = data.find(needle)
        print(f"\nfirst {needle!r} at {idx}: {data[max(0, idx-200): idx+400] if idx != -1 else 'NOT_FOUND'}")
except Exception as e:
    print(f"\nURL probe error: {type(e).__name__}: {e}")
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 268

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- blackbox config ---'
cat -n deployment/observability/blackbox/blackbox.yml

printf '%s\n' '--- search blackbox config files ---'
git ls-files | rg '(^|/)blackbox\.y(?:ml|a)ml$' || true

printf '%s\n' '--- Blackbox HTTP header regexp behavior probe ---'
python3 - <<'PY'
import urllib.request

url = "https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/prober/http.go"
data = urllib.request.urlopen(url, timeout=15).read().decode()

for needle in ["fail_if_header_not_matches", "MatchString", "ReMatch"]:
    idx = data.find(needle)
    print(f"\nfirst {needle!r} at index {idx}")
    if idx != -1:
        start = max(0, idx - 250)
        end = min(len(data), idx + 350)
        print(data[start:end])

print("\n--- MatchString false positives for current rules ---")
import re
patterns = {
    "Upgrade": r'(?i)websocket',
    "Sec-WebSocket-Accept": r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=',
}
cases = {
    "false_positives": ["not-websocket", "websocket-possible", "abcHSmrc0sMlYUkAGmm5OPpG2HaGWk=xyz", "HSmrc0sMlYUkAGmm5OPpG2HaGWk=extra"],
    "expected_true": ["websocket", "HSmrc0sMlYUkAGmm5OPpG2HaGWk="],
}
for header, pat in patterns.items():
    print(f"\nheader {header}: {pat}")
    for label, vals in cases.items():
        hits = [v for v in vals if re.search(pat, v)]
        print(f"  {label}: {hits}")
PY

Repository: ritik4ever/stellar-portfolio-rebalancer

Length of output: 3699

🌐 Web query:

Prometheus blackbox_exporter fail_if_header_not_matches MatchString regex source

💡 Result:

In the Prometheus blackbox_exporter, fail_if_header_not_matches is a configuration block used to fail an HTTP probe if a specific response header does not match a provided regular expression [1][2]. Technical Details and Usage: The fail_if_header_not_matches directive operates on the http_header_match_spec structure [1][3]. This structure requires you to specify a header name, a regular expression, and an optional boolean flag to handle missing headers [4][3]. Key Configuration Parameters: - header: The name of the HTTP header to inspect [4][3]. - regexp: The regular expression string to validate the header value against [4][3]. The blackbox_exporter uses Go's standard library regexp package [4]. - allow_missing: A boolean (defaults to false). If true, the probe will not fail if the header is completely absent from the response [4][3]. If false, the probe will fail if the header is missing [5]. Logic for Multiple Values: When an HTTP response contains multiple values for the same header, fail_if_header_not_matches will succeed (i.e., pass the check) if at least one of the header values matches the provided regular expression [5]. If none of the values match, the probe fails [5]. Example Configuration: The following example from the official documentation demonstrates how to ensure a header matches a specific pattern [2][6]: fail_if_header_not_matches: - header: Access-Control-Allow-Origin regexp: '(*|example.com)' Source Code Implementation: The logic for this check is implemented in the prober/http.go file within the blackbox_exporter repository [5]. It iterates through the values associated with the specified header and uses Regexp.MatchString(val) to perform the validation [5]. If no header values are found and allow_missing is set to false, the probe logs an error and fails [5].

Citations:

Anchor the response header checks and enforce Connection: Upgrade.

Blackbox Exporter uses Regexp.MatchString, so the current unanchored patterns pass false positives like not-websocket or an embedded accept digest. Also add a Connection: Upgrade check as required by the RFC 6455 handshake.

Proposed fix 
       fail_if_header_not_matches:
         - header: Upgrade
-          regexp: (?i)websocket
+          regexp: (?i)^websocket$
+        - header: Connection
+          regexp: (?i)(^|,\s*)Upgrade(\s*,|$)
         - header: Sec-WebSocket-Accept
-          regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
+          regexp: ^HSmrc0sMlYUkAGmm5OPpG2HaGWk=$
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
fail_if_header_not_matches:
- header: Upgrade
regexp: (?i)websocket
- header: Sec-WebSocket-Accept
regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
fail_if_header_not_matches:
- header: Upgrade
regexp: (?i)^websocket$
- header: Connection
regexp: (?i)(^|,\s*)Upgrade(\s*,|$)
- header: Sec-WebSocket-Accept
regexp: ^HSmrc0sMlYUkAGmm5OPpG2HaGWk=$
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/blackbox/blackbox.yml` around lines 25 - 29, Update
the fail_if_header_not_matches checks in the blackbox probe to anchor both
existing regex patterns so they match exactly “websocket” and the expected
Sec-WebSocket-Accept digest, then add a Connection header check requiring the
exact value “Upgrade”.

Source: MCP tools

preferred_ip_protocol: ip4
14 changes: 13 additions & 1 deletion deployment/observability/prometheus/alerts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,22 @@ groups:
for: 5m
labels:
severity: critical
subsystem: api-gateway
service: websocket
annotations:
summary: Backend WebSocket handshake is failing
description: The external WebSocket probe is unable to complete a handshake to the backend.
description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing.

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Make the alert impact statement match the probe’s scope.

A handshake probe does not exercise post-upgrade frame exchange or delivery to already-connected clients. Saying that portfolio and risk pushes “are not reaching connected clients” can mislead responders; describe the observable failure as new WebSocket connections being unable to complete the handshake instead. (rfc-editor.org)

-          description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing.
+          description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend, so new WebSocket connections may fail. Inhibited when BackendDown is already firing.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing.
description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend, so new WebSocket connections may fail. Inhibited when BackendDown is already firing.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/observability/prometheus/alerts.yml` at line 67, Update the
description for the synthetic WebSocket probe alert to state that new WebSocket
connections cannot complete the RFC 6455 opening handshake, rather than claiming
pushes are not reaching already-connected clients. Preserve the existing backend
context and BackendDown inhibition statement.

Source: MCP tools


- alert: WebSocketProbeStalled
expr: absent(probe_success{job="portfolio-websocket", instance="http://backend:3001/"})
for: 10m
labels:
severity: warning
subsystem: api-gateway
service: websocket
annotations:
summary: Synthetic WebSocket probe is not reporting
description: Prometheus has no probe_success sample for the WebSocket endpoint, so availability is currently unmonitored. Check that the blackbox-exporter container is running and reachable.

- alert: Elevated5xxRate
expr: sum(rate(stellar_portfolio_http_requests_total{status_code=~"5.."}[5m])) / clamp_min(sum(rate(stellar_portfolio_http_requests_total[5m])), 1) > 0.05
Expand Down
6 changes: 6 additions & 0 deletions deployment/observability/prometheus/prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,13 @@ scrape_configs:
- target_label: __address__
replacement: blackbox-exporter:9115

# Synthetic uptime probe for the WebSocket endpoint. The backend upgrades any
# non-/ws/portfolio/ path onto the robust broadcast socket, so the root URL is
# the externally reachable WS entrypoint. Probed on its own 30s interval rather
# than the 15s global default to keep handshake churn on the socket low.
- job_name: portfolio-websocket
scrape_interval: 30s
scrape_timeout: 15s
metrics_path: /probe
params:
module: [websocket]
Expand Down
62 changes: 61 additions & 1 deletion docs/OBSERVABILITY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,31 @@ The current deployment probes:

The blackbox configuration is stored in `deployment/observability/blackbox/blackbox.yml`, and Prometheus scrape jobs are defined in `deployment/observability/prometheus/prometheus.yml`.

### Synthetic WebSocket probe

WebSocket availability is measured directly rather than inferred from HTTP metrics. The `websocket` module in `blackbox.yml` issues a real RFC 6455 opening handshake — `Connection: Upgrade`, `Upgrade: websocket`, `Sec-WebSocket-Version: 13` and a fixed `Sec-WebSocket-Key` — and only records success when the server:

1. answers with status `101 Switching Protocols`,
2. echoes an `Upgrade: websocket` response header, and
3. returns the `Sec-WebSocket-Accept` digest derived from the probe's key.

Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line.

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Scope the accept-digest claim to the peer that answered.

RFC 6455 permits reverse proxies and load balancers to offload WebSocket management. A correct Sec-WebSocket-Accept proves that the responding peer processed the key; it does not prove that the backend handled the handshake or that application frames flow end to end. (rfc-editor.org)

-Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line.
+Checking the accept digest matters because a `101` alone only proves that the responding peer agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves that peer processed the probe's key; it does not by itself validate the backend or post-handshake application traffic.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line.
Checking the accept digest matters because a `101` alone only proves that the responding peer agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves that peer processed the probe's key; it does not by itself validate the backend or post-handshake application traffic.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/OBSERVABILITY.md` at line 34, Revise the accept-digest explanation in
the WebSocket observability documentation to state only that a correct
Sec-WebSocket-Accept proves the peer responding to the probe processed its key.
Remove or qualify claims that it proves the backend handled the handshake or
that application frames flow end to end, while retaining the distinction from a
response based solely on status 101.

Source: MCP tools


The probe target is the backend root URL. `backend/src/index.ts` routes every upgrade request that is not under `/ws/portfolio/` onto the robust broadcast socket, so the root URL is the externally reachable WS entrypoint and needs no authentication to complete a handshake.

The `portfolio-websocket` scrape job runs the probe every 30s with a 15s timeout — its own interval rather than the 15s global default, to keep handshake churn on the socket low while still detecting an outage inside one alert evaluation window.

Failures feed the existing Prometheus/Alertmanager pipeline through two rules in `prometheus/alerts.yml`:

| Alert | Fires when | Severity | Route |
| --- | --- | --- | --- |
| `WebSocketHandshakeFailed` | `probe_success == 0` for 5m | critical | pages on-call, suppressed while `BackendDown` is firing |
| `WebSocketProbeStalled` | no `probe_success` sample for 10m | warning | non-paging warnings channel |

`WebSocketProbeStalled` covers the blind spot where the exporter itself is down: without it, a missing probe looks identical to a healthy one.

To probe a deployed environment, add its public WS origin to the `portfolio-websocket` job targets. Use the `http://` or `https://` scheme (not `ws://`) — the blackbox HTTP prober performs the upgrade over an ordinary HTTP request.

## Backend

Backend observability is enabled with environment variables in [backend/.env.example](C:\Users\HP\Documents\students\drips\stellar-portfolio-rebalancer\backend.env.example).
Expand Down Expand Up @@ -119,6 +144,7 @@ Prometheus alerts are preconfigured for:
- backend metrics endpoint down
- backend readiness failures
- frontend uptime failures
- WebSocket handshake failures and a stalled WebSocket probe
- elevated backend 5xx rate
- failed rebalance queue jobs
- stale Reflector price rows observed in the last 15 minutes
Expand All @@ -132,7 +158,41 @@ The backend exports dedicated price-quality metrics:
- `stellar_portfolio_reflector_stale_prices_total`
- `stellar_portfolio_reflector_fallback_usage_total`

Alertmanager ships alerts to `http://host.docker.internal:5001/alerts` by default. Replace that receiver with your Slack, PagerDuty, Opsgenie, or webhook destination before production rollout.
Alertmanager ships non-critical alerts to `http://host.docker.internal:5001/alerts` by default. Replace those receivers with your Slack or webhook destination before production rollout. Critical alerts page the on-call engineer instead — see below.

## On-Call Escalation Policy

Alert routing lives in `deployment/observability/alertmanager/alertmanager.yml`. Severity decides the channel, and only `critical` wakes a human:

| Severity | Receiver | Channel | Group wait | Re-notify |
| --- | --- | --- | --- | --- |
| `critical` | `oncall-pagerduty` | PagerDuty (or Opsgenie) page | 10s | 1h |
| `warning` | `slack-warnings` | Slack, no paging | 30s | 12h |
| `info` | `diagnostic-logs` | Log sink, no paging | 1m | 24h |

### Choosing a vendor

Both a PagerDuty and an Opsgenie receiver are defined. The `severity: critical` route points at `oncall-pagerduty`; switch vendors by changing that route's `receiver` to `oncall-opsgenie`. **Route to one vendor only** — pointing at both would page the on-call engineer twice for every incident.

Credentials are read from files mounted read-only at `/etc/alertmanager/secrets`, backed by `deployment/observability/alertmanager/oncall-secrets/` on the host. See the README in that directory for the file names and setup command; the key files themselves are git-ignored. Without a key file, critical alerts fail to deliver and Alertmanager logs a notification error.

### Avoiding duplicate pages

Three mechanisms keep one incident to one page:

1. **Coarse grouping.** The critical route groups by `subsystem` + `service` rather than `alertname`, so several rules tripping on the same outage land in one notification.
2. **Stable deduplication identity.** Alertmanager derives the PagerDuty `dedup_key` from that group key, and the Opsgenie receiver pins `alias` to `stellar-portfolio-<subsystem>-<service>`. Repeat evaluations update the open incident instead of opening a new one, and the resolved notification closes it.
3. **Inhibition rules.** A firing `critical` alert suppresses `warning` and `info` alerts for the same `subsystem` + `service`. `BackendDown` additionally suppresses the blackbox probe alerts that depend on the backend (`BackendReadinessFailed`, `BackendApiRootFailed`, `ApiDocsProbeFailed`, `WebSocketHandshakeFailed`), since the process being down is their root cause.

### Escalation path

1. **0–10s** — a critical alert fires; Alertmanager holds it for `group_wait` to collect related alerts into the same page.
2. **10s** — the page reaches the primary on-call engineer. The payload carries `alertname`, `subsystem`, `service`, the firing-alert count, and a link back to this document.
3. **Acknowledge and triage.** Cross-check the matching Sentry release and environment tags first — that narrows the search to the exact build that produced the failure. See [TRIAGE.md](TRIAGE.md).
4. **1h unacknowledged** — Alertmanager re-notifies (`repeat_interval: 1h`). Configure secondary-responder escalation in the PagerDuty/Opsgenie escalation policy itself, not here; Alertmanager only delivers the page.
5. **Resolution.** Alertmanager sends a resolve notification (`send_resolved: true`) and the incident closes automatically when the underlying alert stops firing.

Adding a new critical alert requires `severity: critical` plus `subsystem` and `service` labels. Without those two labels the alert still pages, but it groups on its own and cannot be inhibited by a related root-cause alert.

### Queue Operations Dashboard

Expand Down
Loading