feat(observability): add on-call paging receivers and synthetic WebSo… - #1614
Conversation
…cket probe Alertmanager previously named a receiver `pagerduty-critical` but pointed it at a local webhook, so nothing actually paged. Replace it with real integrations and make the WebSocket endpoint's availability directly measured rather than inferred. On-call paging (ritik4ever#1492): - Add `oncall-pagerduty` (Events API v2) and `oncall-opsgenie` receivers, with credentials read from files mounted at /etc/alertmanager/secrets so keys stay out of version control. Route critical alerts to PagerDuty; switching vendors is a one-line receiver change. - Leave warning and info alerts on their existing non-paging receivers. - Suppress duplicate pages three ways: group the critical route by subsystem+service instead of alertname, pin the Opsgenie alias to that same incident identity (PagerDuty derives dedup_key from the group key), and add inhibit rules so a critical alert silences same-service warnings and BackendDown silences the probe alerts it causes. Also relax the critical group_interval from 1m to 5m. Synthetic WebSocket probe (ritik4ever#1494): - Verify the blackbox handshake properly: assert the response echoes `Upgrade: websocket` and returns the Sec-WebSocket-Accept digest derived from the probe key, so a bare 101 from a proxy no longer counts as success. - Give the portfolio-websocket job its own 30s interval and 15s timeout instead of inheriting the 15s global scrape interval. - Label WebSocketHandshakeFailed with `subsystem: api-gateway` so it groups and inhibits alongside the other api-gateway alerts, and add WebSocketProbeStalled to catch an absent probe — otherwise a dead exporter looks healthy. Document the escalation policy, dedup mechanisms, and probe behaviour in docs/OBSERVABILITY.md. Closes ritik4ever#1492 Closes ritik4ever#1494 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@manoahLinks is attempting to deploy a commit to the ritik4ever's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
@manoahLinks Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
📝 WalkthroughWalkthroughThe PR adds PagerDuty/Opsgenie Alertmanager integrations with secret-file credentials, critical-alert inhibition and grouping, plus synthetic WebSocket probing through Blackbox Exporter, Prometheus scraping, new alerts, and observability documentation. ChangesOn-call alerting
Synthetic WebSocket monitoring
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deployment/observability/alertmanager/alertmanager.yml`:
- Around line 85-89: Update the severity inhibition rule in Alertmanager so
queue or other unlabeled critical alerts cannot inhibit unrelated warning/info
alerts: ensure all affected alert definitions provide non-empty subsystem and
service labels, or constrain both source_matchers and target_matchers to require
non-empty values for subsystem and service while retaining them in equal.
- Around line 125-131: Remove the unsupported alias entry from the
oncall-opsgenie opsgenie_configs configuration, leaving the api_key_file,
priority, message, and description settings unchanged.
In `@deployment/observability/alertmanager/oncall-secrets/README.md`:
- Around line 15-17: Update the secret setup instructions in the README so the
bind-mounted pagerduty_routing_key is readable by Alertmanager’s container user
(UID 65534) while remaining safely restricted. Document applying UID 65534
ownership or an equivalent compatible group/permissions after creating the file,
and keep the routing_key_file path unchanged.
In `@deployment/observability/blackbox/blackbox.yml`:
- Around line 10-12: Update deployment/observability/blackbox/blackbox.yml lines
13-29 to use a Blackbox Exporter version supporting websocket_probe, with
prober: websocket and websocket_probe configuration instead of the HTTP upgrade
probe. Update docs/OBSERVABILITY.md lines 28-32 and 49 to describe the WebSocket
prober and remove claims that the HTTP prober completes the handshake.
- Around line 25-29: Update the fail_if_header_not_matches checks in the
blackbox probe to anchor both existing regex patterns so they match exactly
“websocket” and the expected Sec-WebSocket-Accept digest, then add a Connection
header check requiring the exact value “Upgrade”.
In `@deployment/observability/prometheus/alerts.yml`:
- Line 67: Update the description for the synthetic WebSocket probe alert to
state that new WebSocket connections cannot complete the RFC 6455 opening
handshake, rather than claiming pushes are not reaching already-connected
clients. Preserve the existing backend context and BackendDown inhibition
statement.
In `@docs/OBSERVABILITY.md`:
- Line 34: Revise the accept-digest explanation in the WebSocket observability
documentation to state only that a correct Sec-WebSocket-Accept proves the peer
responding to the probe processed its key. Remove or qualify claims that it
proves the backend handled the handshake or that application frames flow end to
end, while retaining the distinction from a response based solely on status 101.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 069e1058-dcee-48c4-9bbf-a2013b69f0c1
📒 Files selected for processing (8)
deployment/docker-compose.ymldeployment/observability/alertmanager/alertmanager.ymldeployment/observability/alertmanager/oncall-secrets/.gitignoredeployment/observability/alertmanager/oncall-secrets/README.mddeployment/observability/blackbox/blackbox.ymldeployment/observability/prometheus/alerts.ymldeployment/observability/prometheus/prometheus.ymldocs/OBSERVABILITY.md
| - source_matchers: | ||
| - severity = critical | ||
| target_matchers: | ||
| - severity =~ "warning|info" | ||
| equal: ['subsystem', 'service'] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
# Verify every alert participating in severity-based routing supplies both
# correlation labels before relying on the inhibition rule.
rg -n -C 6 'severity:|subsystem:|service:' deployment/observability/prometheus/alerts.ymlRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 15704
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== alertmanager config relevant sections =="
sed -n '1,130p' deployment/observability/alertmanager/alertmanager.yml
echo
echo "== alert rules missing subsystem/service label occurrences =="
python3 - <<'PY'
from pathlib import Path
import re
path=Path('deployment/observability/prometheus/alerts.yml')
text=path.read_text()
# naive rule object by indentation: - alert: starts, labels block may exist
rules=[]
cur=None
for line in text.splitlines():
if re.match(rb'^\s+-\s+alert:\s+\S+', line):
cur=''
if cur is not None:
cur += line + '\n'
if re.match(rb'^\s+-\s+alert:\s+\S+', line) and len(cur)>5000:
cur=None
rules=[]
blocks=[]
depth=None
block=[]
for line in text.splitlines(keepends=True):
stripped=line.rstrip()
# simple Prometheus rule-level indent after name interval: alert: is 6 spaces in this file
if re.match(r'^\s{6}-\s+alert:', line):
if block:
blocks.append(''.join(block))
block=[line]
elif block:
indent=len(line) - len(line.lstrip())
if len(line.strip())==0:
block.append(line)
continue
if re.match(r'^\s{6}-\s+alert:', line):
# another rule same indentation
blocks.append(''.join(block))
block=[line]
elif re.match(r'^\s{0,5}-\s+', line):
pass
else:
block.append(line)
if block: blocks.append(''.join(block))
missing=[]
for b in blocks:
m = re.search(r'^\s{6}-\s+alert:\s+(\S+)', b, re.M)
name = m.group(1) if m else '<unknown>'
labels = re.search(r'^\s*labels:\s*\n((?:[^\n]+(?:\n|$))*)', b, re.M)
if not labels:
missing.append((name, 'no labels block'))
continue
txt = labels.group(1)
if not re.search(r'^\s*subsystem:', txt, re.M):
if not re.search(r'^\s*service:', txt, re.M):
missing.append((name, 'no subsystem/service labels'))
elif re.search(r'^\s*severity:(?:\s+critical|warning|info|debug)?', txt, re.M):
missing.append((name, 'no subsystem label'))
elif not re2.search(r'^\s*service:', txt, re.M):
missing.append((name, 'no service label'))
PY
echo
echo "== receiver templates mentioning label substitution/fallback =="
rg -n 'template=|{{\[.*\]|{{ \(.*\)|subsystem|service|severity' deployment/observability/alertmanager deployment/observability -g '*.tmpl' -g '*.yml' | head -200Repository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 5622
🌐 Web query:
Alertmanager inhibit rule missing labels treated as equal documentation
💡 Result:
In Alertmanager, inhibition rules treat a missing label and a label with an empty value as semantically equivalent [1][2][3]. Consequently, if all label names listed in the equal field of an inhibition rule are absent from both the source and target alerts, the inhibition rule will consider them to match and will apply [1][2]. This behavior can lead to unintended, global suppression if you are not careful, as alerts missing the specified labels may be inhibited by any firing source alert that also lacks those labels [4]. To avoid this "missing equals missing" behavior and ensure that inhibition only occurs when labels are explicitly present and matching, you can use a regular expression matcher in your source or target matchers to enforce that the label must have a non-empty value [5][6]. For example: inhibit_rules: - source_matchers: - alertname = SourceAlert target_matchers: - alertname = TargetAlert - equal_label =~ ".+" # Ensures the label is present and not empty equal: - equal_label This pattern prevents the inhibition rule from applying if the label is missing, as the requirement for a non-empty value will not be satisfied [5]. Alternatively, simply ensuring that your alerting and recording rules consistently attach all required labels can mitigate this issue [4].
Citations:
- 1: https://prometheus.io/docs/alerting/latest/configuration/
- 2: https://github.com/prometheus/alertmanager/blob/master/docs/configuration.md
- 3: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/inhibition-rules/
- 4: https://oneuptime.com/blog/post/2026-01-27-alertmanager-inhibition-rules/view
- 5: Inhibition suppresses alerts missing the defined "equal" labels prometheus/alertmanager#1727
- 6: Inhibitor incorrectly inhibits alerts where no 'equal' labels are present prometheus/alertmanager#507
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import re
from pathlib import Path
text = Path('deployment/observability/prometheus/alerts.yml').read_text()
blocks = re.split(r'(?=^\s{6}-\s+alert:)', text, flags=re.M)
missing = []
for b in blocks:
m = re.search(r'^\s{6}-\s+alert:\s+(\S+)', b, re.M)
if not m:
continue
name = m.group(1)
labels = re.search(r'^\s*labels:\s*\n((?:[^\n]+(?:\n|$))*)', b, re.M)
if not labels:
missing.append((name, 'no labels block'))
continue
txt = labels.group(1)
has_subsystem = bool(re.search(r'^\s*subsystem:', txt, re.M))
has_service = bool(re.search(r'^\s*service:', txt, re.M))
if not has_subsystem or not has_service:
missing.append((name, f'missing subsystem={has_subsystem}/service={has_service}'))
print('alert rules missing required correlation labels:')
for rule, why in missing:
print(f'- {rule}: {why}')
# Simulate Alertmanager missing-label inhibition equality for this rule.
# A missing label without an explicit regex matcher is considered equal to
# another missing label with the same name.
examples = [
({"severity": "critical"}, {"severity": "warning"}),
({"severity": "critical"}, {"severity": "info"}),
({"severity": "critical", "subsystem": "api-gateway"}, {"severity": "warning", "service": "backend"}),
({"severity": "critical"}, {"severity": "warning", "subsystem": "api-gateway"}),
]
for src, tgt in examples:
equal = ['subsystem', 'service']
miss_src = all(k not in src for k in equal)
miss_tgt = all(k not in tgt for k in equal)
matched = miss_src and miss_tgt
print(f'{src} | {tgt} -> inhibit={matched}')
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 1511
Do not let queue labels alone determine inhibition scope.
Several severity-routed rules are missing subsystem (BackendApiRootFailed, ApiDocsProbeFailed, and the queue alerts) while others are missing service. Since Alertmanager treats missing labels as equal, an unlabeled critical queue alert could suppress unrelated warnings/info alerts that also lack both required labels. Require both correlation labels in alert definitions, or add non-empty regex constraints to both source and target matchers plus equal.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/alertmanager/alertmanager.yml` around lines 85 - 89,
Update the severity inhibition rule in Alertmanager so queue or other unlabeled
critical alerts cannot inhibit unrelated warning/info alerts: ensure all
affected alert definitions provide non-empty subsystem and service labels, or
constrain both source_matchers and target_matchers to require non-empty values
for subsystem and service while retaining them in equal.
| - name: oncall-opsgenie | ||
| opsgenie_configs: | ||
| - api_key_file: /etc/alertmanager/secrets/opsgenie_api_key | ||
| priority: P1 | ||
| alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}' | ||
| message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})' | ||
| description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}' |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
docker run --rm \
-v "$PWD/deployment/observability/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro" \
prom/alertmanager:v0.27.0 \
amtool check-config /etc/alertmanager/alertmanager.ymlRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 221
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== alertmanager config excerpt =="
sed -n '100,145p' deployment/observability/alertmanager/alertmanager.yml
echo
echo "== referenced config.go alias handling from prometheus/alertmanager v0.27.0 =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/config/config.go | sed -n '800,950p'
echo
echo "== occurrences of operator.alias in v0.27.0 source =="
curl -fsSL https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1 \
| python3 - <<'PY'
import json, sys
data=json.load(sys.stdin)
tree=json.loads(data['message'])['tree']
for item in tree:
if item['type'] == 'blob' and item['path'].endswith(('.go','.yaml','.yml')):
print(item['path'])
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 9153
🌐 Web query:
prometheus alertmanager OpsgenieConfig alias field config.go v0.27.0
💡 Result:
In Prometheus Alertmanager v0.27.0, there is no configurable alias field available in the OpsGenieConfig definition within config.go [1][2][3]. Alertmanager automatically generates the alias field for Opsgenie notifications to facilitate deduplication [4][5]. This alias is derived internally based on the grouping labels (group_by) configured for the specific route in Alertmanager [5]. Because this process is handled automatically within the notification pipeline, users cannot explicitly set or override this field via the configuration file [1][5][6]. Attempts to customize this behavior, such as preventing cross-team deduplication, have historically been addressed via workarounds rather than direct configuration, including: 1. Using external_labels in Prometheus to ensure that labels used in Alertmanager's group_by are unique per cluster or environment [4]. 2. Configuring a prefix on the Opsgenie side within the Prometheus Integration settings (if supported by your Opsgenie plan), which adds a prefix to the alias field for all incoming alerts for that integration [7]. As of v0.27.0, there have been no changes to OpsGenieConfig to introduce an alias field, and feature requests for such functionality have been noted in the project's issue tracker [4][8][7].
Citations:
- 1: https://github.com/prometheus/alertmanager/blob/main/notify/opsgenie/opsgenie.go
- 2: https://github.com/prometheus/alertmanager/blob/master/config/config.go
- 3: https://github.com/prometheus/alertmanager/blob/v0.25.0/config/notifiers.go
- 4: Add option to customize alias field for opsgenie alerts prometheus/alertmanager#1598
- 5: Alert reach Opsgenie with different deduplicating aliases even in the presence of group_by labels prometheus/alertmanager#3328
- 6: https://github.com/prometheus/alertmanager/blob/master/notify/opsgenie/opsgenie.go
- 7: OpsGenie cross-team deduplication - alias prefix prometheus/alertmanager#3639
- 8: https://github.com/prometheus/alertmanager/blob/v0.27.0/CHANGELOG.md
🌐 Web query:
alertmanager config validation unknown fields error OpsGenieConfig alias
💡 Result:
The "unknown field" error in Alertmanager configuration, particularly when involving OpsGenie or other receiver configurations, typically stems from one of two primary causes: 1. Using Native Alertmanager Syntax in a Kubernetes AlertmanagerConfig CRD The most common cause for "unknown field" errors when using the Prometheus Operator is providing raw Alertmanager configuration (which uses snake_case, e.g., opsgenie_configs, webhook_configs) inside a Kubernetes AlertmanagerConfig Custom Resource Definition (CRD) [1][2]. The AlertmanagerConfig CRD has its own schema (which uses camelCase, e.g., opsgenieConfigs, webhookConfigs) [3][1][2]. - Solution: Ensure your YAML structure conforms to the AlertmanagerConfig CRD specification rather than the native Alertmanager config file format [1][2]. You can verify the expected fields for your specific version using the command kubectl explain alertmanagerconfig.spec.receivers [1]. 2. Feature Mismatch Between Operator and Alertmanager If you are configuring the native Alertmanager configuration file (often via a Kubernetes Secret or Helm values) and receive an "unknown field" error, your Prometheus Operator or Alertmanager version may be outdated and not yet support the specific field you are trying to use [4][5]. - Solution: Verify that both your Prometheus Operator and Alertmanager image versions support the desired configuration field [4][5]. Features like new receiver types or configuration options (e.g., api_key_file) are added incrementally; if the operator manages the config, ensure it is updated to a version that recognizes the new fields [4][5]. Regarding the specific term "alias" in your query: "alias" is not a standard field within the native Alertmanager opsgenie_config schema [6]. If you are encountering an error mentioning "alias" as an unknown field, it is likely an unsupported or incorrectly placed configuration key. If you are attempting to alias an OpsGenie configuration, verify the current Alertmanager documentation to see if that functionality is supported or if it must be handled differently (e.g., using different receiver names) [6].
Citations:
- 1: https://exchangetuts.com/index.php/prometheus-operator-alertmanagerconfig-unknown-field-webhook-configs-1766166902941814
- 2: https://stackoverflow.com/questions/66418893/prometheus-operator-alertmanagerconfig-unknown-field-webhook-configs
- 3: https://docs.okd.io/latest/rest_api/monitoring_apis/alertmanagerconfig-monitoring-coreos-com-v1beta1.html
- 4: opsgenie_configs field api_key_file not found in type config.plain prometheus-operator/prometheus-operator#4714
- 5: AlertManagerConfig: Add support for the new msteamsv2_configs receiver prometheus-operator/prometheus-operator#7051
- 6: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/alertmanager.yml.example \
| sed -n '1,180p'
echo
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/config/notifiers.go \
| sed -n '480,580p'
echo
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/notify/opsgenie/opsgenie.go \
| sed -n '1,180p'Repository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 228
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== candidate config/notifier files in v0.27.0 =="
python3 - <<'PY'
import json, urllib.request
url='https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1'
try:
with urllib.request.urlopen(url) as r:
data=json.load(r)
except Exception as e:
print('GH API payload error:', repr(e))
raise SystemExit(0)
for item in (data.get('tree') or []):
if item.get('type')=='blob' and item.get('path') in {'config/notifiers.go','notify/opsgenie/opsgenie.go','config/config.go'}:
print(item['path'], item.get('sha'))
PY
echo
curl -fsSL https://github.com/prometheus/alertmanager/raw/v0.27.0/config/notifiers.go | sed -n '1,320p' | sed -n '440,560p'
echo
curl -fsSL https://github.com/prometheus/alertmanager/raw/v0.27.0/notify/opsgenie/opsgenie.go | sed -n '80,220p'Repository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 4238
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import ssl, urllib.request
ctx=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname=False
ctx.verify_mode=ssl.CERT_NONE
data=json.load(urllib.request.urlopen('https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1', context=ctx))
for item in data.get('tree', []):
if item.get('path') in {'config/notifiers.go','config/config.go'}:
print('FETCHING', item.get('path'), item.get('sha'))
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 288
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import json, ssl, urllib.request
ctx=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname=False
ctx.verify_mode=ssl.CERT_NONE
data=json.load(urllib.request.urlopen('https://api.github.com/repos/prometheus/alertmanager/git/trees/v0.27.0?recursive=1', context=ctx))
for item in data.get('tree', []):
if item.get('path') in {'config/notifiers.go','config/config.go'}:
print('FETCHING', item.get('path'), item.get('sha'))
PY
echo
curl -fsSL --insecure https://github.com/prometheus/alertmanager/raw/v0.27.0/config/notifiers.go | sed -n '420,540p'
echo
curl -fsSL --insecure https://github.com/prometheus/alertmanager/raw/v0.27.0/config/config.go | rg -n "OpsGenieConfig|alias|yaml:" -C 2 | sed -n '1,220p'Repository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 973
Remove the unsupported Opsgenie alias field.
opsgenie_configs.alias is not part of Alertmanager v0.27.0’s Opsgenie schema, and that notifier sets Opsgenie alias to the Alertmanager group key (key.Hash()), so this unknown field prevents config load.
Proposed fix
- # Opsgenie equivalent of the receiver above. `alias` pins the deduplication
- # identity to the incident (subsystem + service) so Opsgenie keeps appending to
- # the open alert rather than creating a new one per evaluation cycle.
+ # Opsgenie uses Alertmanager's group key for its alias, preserving
+ # deduplication for this route's subsystem/service grouping.
- name: oncall-opsgenie
opsgenie_configs:
- api_key_file: /etc/alertmanager/secrets/opsgenie_api_key
priority: P1
- alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: oncall-opsgenie | |
| opsgenie_configs: | |
| - api_key_file: /etc/alertmanager/secrets/opsgenie_api_key | |
| priority: P1 | |
| alias: 'stellar-portfolio-{{ or .CommonLabels.subsystem "platform" }}-{{ or .CommonLabels.service "unknown" }}' | |
| message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})' | |
| description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}' | |
| - name: oncall-opsgenie | |
| opsgenie_configs: | |
| - api_key_file: /etc/alertmanager/secrets/opsgenie_api_key | |
| priority: P1 | |
| message: '{{ .CommonLabels.alertname }} ({{ or .CommonLabels.subsystem "platform" }})' | |
| description: '{{ or .CommonAnnotations.description .CommonAnnotations.summary }}' |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/alertmanager/alertmanager.yml` around lines 125 -
131, Remove the unsupported alias entry from the oncall-opsgenie
opsgenie_configs configuration, leaving the api_key_file, priority, message, and
description settings unchanged.
| ```bash | ||
| printf '%s' "$PAGERDUTY_ROUTING_KEY" > pagerduty_routing_key | ||
| chmod 600 pagerduty_routing_key |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
docker image inspect prom/alertmanager:v0.27.0 --format '{{.Config.User}}'
stat -c '%u:%g %a %n' \
deployment/observability/alertmanager/oncall-secrets/pagerduty_routing_keyRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 221
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== files =="
git ls-files | rg 'deployment/observability/alertmanager|oncall-secrets|README|compose|docker|Dockerfile' || true
echo
echo "== target README =="
if [ -f deployment/observability/alertmanager/oncall-secrets/README.md ]; then
cat -n deployment/observability/alertmanager/oncall-secrets/README.md
fi
echo
echo "== alertmanager compose/service references =="
rg -n "alertmanager|PAGERDUTY_ROUTING_KEY|pagerduty_routing_key|security_context|users|uid|gid|user:" -S deployment .github 2>/dev/null || true
echo
echo "== upstream alertmanager v0.27.0 Dockerfile relevant lines =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | sed -n '1,220p' | nl -baRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 8307
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== compose alertmanager block =="
sed -n '180,210p' deployment/docker-compose.yml | cat -v
echo
echo "== alertmanager config relevant routing/receiver =="
sed -n '90,135p' deployment/observability/alertmanager/alertmanager.yml
echo
echo "== upstream v0.27.0 Dockerfile lines =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | awk 'NR<=220{printf "%4d %s\n", NR, $0}'
echo
echo "== parse dockerfile user directives =="
curl -fsSL https://raw.githubusercontent.com/prometheus/alertmanager/v0.27.0/Dockerfile | awk 'toupper($1)=="USER"{users=users (users?"\\n":"") NR": "$0} END{print (users?users:"NO USER DIRECTIVE FOUNDED")}'
echo
echo "== static permissions probe for current README command behavior =="
tmp="$(mktemp -d)"
cd "$tmp"
USER="$(id -u):$(id -g)"
PAGERDUTY_ROUTING_KEY=fake-key
printf '%s' "$PAGERDUTY_ROUTING_KEY" > pagerduty_routing_key
chmod 600 pagerduty_routing_key
stat -c 'mode=%a owner=%u:%g file=%n' pagerduty_routing_key
if [ "$(stat -c '%u' pagerduty_routing_key)" = "$(id -u)" ]; then
echo "current_host_user_matches_file_owner=true"
else
echo "current_host_user_matches_file_owner=false"
fiRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 4437
🌐 Web query:
Docker USER nobody UID GID nobody container filesystem permissions mode 0600 readable by nobody
💡 Result:
In a Docker container, a file with 0600 (read/write only for the owner) permissions is readable by the nobody user if and only if that user is the actual owner of the file [1][2]. The nobody user (typically UID 65534) is not a special "superuser" that bypasses standard Unix filesystem permissions; it is simply a restricted, low-privilege account [3][4][5]. When you encounter a situation where a container running as nobody cannot read a 0600 file, it is due to one of the following factors: 1. Ownership Mismatch: If the file is owned by a different UID (even if the usernames match, as UID/GID numbers are what the Linux kernel enforces), the nobody user will be denied access [1][6]. 2. User Namespaces (User Remapping): If you are using rootless Docker or user namespace remapping (userns-remap), the UIDs inside the container are shifted relative to the host [7][6]. A nobody user (UID 65534) inside the container is mapped to a different, high-range UID on the host [4]. If the file on the host is owned by the host's actual nobody user, the container's nobody user will not have access because the kernel sees them as different identities [7][6]. 3. Filesystem Limitations: Docker bind mounts do not automatically translate or map permissions between the host and container [8][1]. The container process must have the appropriate UID/GID match or sufficient permissions (e.g., group or world read access) to interact with the file [9][2]. To resolve access issues for 0600 files: - Ensure the effective UID/GID of the container process matches the UID/GID of the file owner on the host [9][1]. - If you cannot change the process UID, adjust the file permissions on the host to be group or world-readable (e.g., 0640 or 0644), provided this aligns with your security requirements [10][2]. - When using user namespaces, ensure the host-side file ownership corresponds to the remapped UID that the container's nobody user resolves to on the host [6].
Citations:
- 1: https://www.dash0.com/faq/how-to-manage-permissions-for-docker-shared-volumes
- 2: [Bug] Inbound media files saved with 0600 permissions — unreadable from Docker sandbox openclaw/openclaw#17941
- 3: https://www.docker.com/blog/understanding-the-docker-user-instruction/
- 4: Rootless Docker Volume Permissions Issue with User Namespace Remapping and ACLs moby/moby#51238
- 5: Source secret file needs world readable permission docker/compose#7436
- 6: https://stackoverflow.com/questions/77655209/rootless-docker-host-file-permissions
- 7: https://stackoverflow.com/questions/68524804/docker-run-maps-volumes-with-user-nobody-of-nogroup
- 8: Docker Compose
userDirective Causes Permission Failures When UID Differs from Image's Built-in User docker/compose#13758 - 9: https://stackoverflow.com/questions/71918710/allow-docker-container-host-user-to-write-on-bind-mounted-host-directory
- 10: https://unix.stackexchange.com/questions/523492/how-to-mount-files-as-specific-user-when-using-docker-namespace-remapping
Make the bind-mounted secret readable by Alertmanager.
prom/alertmanager:v0.27.0 runs as the container nobody UID, while this command creates pagerduty_routing_key owned by the host user with 0600. Alertmanager then cannot read routing_key_file: /etc/alertmanager/secrets/pagerduty_routing_key, and critical PagerDuty notifications fail. Document setup using UID 65534 ownership/a compatible group, or set container permissions that expose the secret safely.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/alertmanager/oncall-secrets/README.md` around lines
15 - 17, Update the secret setup instructions in the README so the bind-mounted
pagerduty_routing_key is readable by Alertmanager’s container user (UID 65534)
while remaining safely restricted. Document applying UID 65534 ownership or an
equivalent compatible group/permissions after creating the file, and keep the
routing_key_file path unchanged.
| # Synthetic WebSocket probe. Sends a real RFC 6455 opening handshake and only | ||
| # reports success when the server completes the protocol switch, so a plain | ||
| # HTTP response from a partially-started backend still counts as a failure. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^deployment/observability/blackbox/blackbox.yml$|^docs/OBSERVABILITY.md$|^deploy|^blackbox|observability|package|requirements|Makefile|README)'
echo
echo "== blackbox config =="
if [ -f deployment/observability/blackbox/blackbox.yml ]; then
cat -n deployment/observability/blackbox/blackbox.yml
fi
echo
echo "== observability docs relevant =="
if [ -f docs/OBSERVABILITY.md ]; then
nl -ba docs/OBSERVABILITY.md | sed -n '1,90p'
fi
echo
echo "== exporter version / pin refs =="
rg -n "blackbox|blackbox_exporter|portworx|websocket|portfolio-websocket|prober:" . \
--glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || trueRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 3695
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== observability docs relevant =="
sed -n '1,90p' docs/OBSERVABILITY.md
echo
echo "== exporter / scrape references =="
sed -n '1,220p' deployment/observability/prometheus/prometheus.yml
echo
sed -n '1,160p' .github/workflows/observability-lint.yml
echo
rg -n "blackbox|blackbox_exporter|websocket|portfolio-websocket|prober:" deployment deploy README*.md docs .github --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || true
echo
echo "== scripts observability references =="
sed -n '1,220p' scripts/bootstrap-observability.sh 2>/dev/null || trueRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 22393
🌐 Web query:
Prometheus Blackbox Exporter websocket prober PROTOCOL error HTTP responses 101 Connection upgrade documentation
💡 Result:
As of July 30, 2026, Prometheus Blackbox Exporter supports WebSocket probing directly through a dedicated websocket prober [1][2]. If you are encountering issues with HTTP 101 (Switching Protocols) responses during probing, note the following: 1. Dedicated WebSocket Prober: The websocket prober was specifically introduced to handle the WebSocket handshake and connection upgrade process [1]. Ensure you are using prober: websocket in your configuration file rather than the standard http prober, as the http prober is not designed to maintain or interact with an upgraded WebSocket connection [2][3]. 2. Configuration: The websocket prober configuration allows you to define a query_response sequence [2]. This includes an expect field (which uses regular expressions to match incoming server messages) and a send field (to send messages after the connection is successfully upgraded) [2]. 3. Troubleshooting 101 Responses: * The HTTP 101 Switching Protocols status code is the expected successful response for a WebSocket upgrade request [4]. * If your probe is failing, use the debug=true parameter in your probe URL (e.g., /probe?target=...&module=...&debug=true) to view detailed logs of the request and response lifecycle [5]. This will confirm if the server is correctly returning the 101 status and if the handshake is completing successfully [5]. * Verify that your server implementation strictly adheres to the WebSocket handshake requirements defined in RFC 6455, as the client will drop the connection if the protocol switch is not handled correctly [6][4]. For detailed configuration syntax, refer to the official Blackbox Exporter documentation [2]. Ensure your version of Blackbox Exporter is up to date, as this feature was finalized and merged into the main codebase in early 2026 [1].
Citations:
- 1: feat: Add websocket connection prober support prometheus/blackbox_exporter#1278
- 2: https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
- 3: https://github.com/prometheus/blackbox_exporter/blob/bee8e9102a106bff63281ee9c64c7b1275ef21d0/CONFIGURATION.md
- 4: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/101
- 5: https://groups.google.com/g/prometheus-users/c/1a2K7a_ZIGc
- 6: https://www.http-probe.com/websockets/
🌐 Web query:
Prometheus Blackbox Exporter v0.25.0 websocket prober configuration reference
💡 Result:
The Prometheus Blackbox Exporter version 0.25.0, released on April 9, 2024, does not include support for a websocket prober [1][2]. WebSocket support was added to the Blackbox Exporter project after the release of v0.25.0 [3]. The feature was introduced via pull request #1278 [3]. If you require WebSocket probing capabilities, you must use a more recent version of the Blackbox Exporter that includes this feature [4][3]. When using a version that supports the websocket prober (e.g., current main branch), the configuration structure is as follows: websocket_probe: # Optional HTTP request configuration http_config: [ <http_client_config> ] # The HTTP headers set for the probe headers: [ : [ values: [,...] ], [ secrets: [,...] ], [ files: [,...] ] ],... # The query sent after connection upgrade and the expected associated response query_response: [ - [ expect: ], [ send: ], [ starttls: <boolean | default = false> ] ],... # The IP protocol of the Websocket probe (ip4, ip6) [ preferred_ip_protocol: | default = "ip6" ] # Fallback to the other IP protocol if the preferred_ip_protocol fails [ ip_protocol_fallback: | default = true ] For examples of how to define this in your configuration file, refer to the official example.yml found in the project repository [5].
Citations:
- 1: https://github.com/prometheus/blackbox_exporter/releases/tag/v0.25.0
- 2: https://github.com/prometheus/blackbox_exporter/blob/135d228d/CHANGELOG.md
- 3: feat: Add websocket connection prober support prometheus/blackbox_exporter#1278
- 4: https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
- 5: https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/example.yml
Use Blackbox Exporter websocket_probe, not an HTTP upgrade probe.
prom/blackbox-exporter:v0.25.0 does not support a prober: websocket, so this http module only performs a one-shot 101 response check and falls back to draining until closed/timeout. Either update Blackbox to a version with websocket_probe support and change:
deployment/observability/blackbox/blackbox.yml#L13-L29toprober: websocket/websocket_probe:configuration, anddocs/OBSERVABILITY.mdto stop saying the HTTP prober completes the WebSocket handshake.
📍 Affects 2 files
deployment/observability/blackbox/blackbox.yml#L10-L12(this comment)docs/OBSERVABILITY.md#L28-L32docs/OBSERVABILITY.md#L49-L49
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/blackbox/blackbox.yml` around lines 10 - 12, Update
deployment/observability/blackbox/blackbox.yml lines 13-29 to use a Blackbox
Exporter version supporting websocket_probe, with prober: websocket and
websocket_probe configuration instead of the HTTP upgrade probe. Update
docs/OBSERVABILITY.md lines 28-32 and 49 to describe the WebSocket prober and
remove claims that the HTTP prober completes the handshake.
Source: MCP tools
| fail_if_header_not_matches: | ||
| - header: Upgrade | ||
| regexp: (?i)websocket | ||
| - header: Sec-WebSocket-Accept | ||
| regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk= |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python - <<'PY'
import re
assert re.search(r'(?i)websocket', 'not-websocket')
assert re.fullmatch(r'(?i)websocket', 'not-websocket') is None
assert re.fullmatch(r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=', 'HSmrc0sMlYUkAGmm5OPpG2HaGWk=')
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 177
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '--- blackbox config ---\n'
cat -n deployment/observability/blackbox/blackbox.yml
printf '\n--- search blackbox config files ---\n'
git ls-files | rg '(^|/)blackbox\.y(?:ml|a)ml$' || true
printf '\n--- Prometheus/blackbox_exporter behavior probe from source/docs if present in repo or curl upstream raw ---\n'
python3 - <<'PY'
import re
import urllib.request
patterns = {
"unanchored_upgrade": r'(?i)websocket',
"unanchored_accept": r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=',
}
cases = {
"false_positives": ["not-websocket", "websocket-possible", "abcHSmrc0sMlYUkAGmm5OPpG2HaGWk=xyz", "HSmrc0sMlYUkAGmm5OPpG2HaGWk=extra"],
"expected_true": ["websocket", "HSmrc0sMlYUkAGmm5OPpG2HaGWk="],
}
for name, pat in patterns.items():
print(f"\n{name}: {pat}")
for label, vals in cases.items():
hits = [v for v in vals if re.search(pat, v)]
print(f" {label}: {hits}")
url = "https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/prober/http.go"
try:
data = urllib.request.urlopen(url, timeout=15).read().decode()
for needle in ["fail_if_header_not_matches", "MatchString"]:
idx = data.find(needle)
print(f"\nfirst {needle!r} at {idx}: {data[max(0, idx-200): idx+400] if idx != -1 else 'NOT_FOUND'}")
except Exception as e:
print(f"\nURL probe error: {type(e).__name__}: {e}")
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 268
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- blackbox config ---'
cat -n deployment/observability/blackbox/blackbox.yml
printf '%s\n' '--- search blackbox config files ---'
git ls-files | rg '(^|/)blackbox\.y(?:ml|a)ml$' || true
printf '%s\n' '--- Blackbox HTTP header regexp behavior probe ---'
python3 - <<'PY'
import urllib.request
url = "https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/prober/http.go"
data = urllib.request.urlopen(url, timeout=15).read().decode()
for needle in ["fail_if_header_not_matches", "MatchString", "ReMatch"]:
idx = data.find(needle)
print(f"\nfirst {needle!r} at index {idx}")
if idx != -1:
start = max(0, idx - 250)
end = min(len(data), idx + 350)
print(data[start:end])
print("\n--- MatchString false positives for current rules ---")
import re
patterns = {
"Upgrade": r'(?i)websocket',
"Sec-WebSocket-Accept": r'HSmrc0sMlYUkAGmm5OPpG2HaGWk=',
}
cases = {
"false_positives": ["not-websocket", "websocket-possible", "abcHSmrc0sMlYUkAGmm5OPpG2HaGWk=xyz", "HSmrc0sMlYUkAGmm5OPpG2HaGWk=extra"],
"expected_true": ["websocket", "HSmrc0sMlYUkAGmm5OPpG2HaGWk="],
}
for header, pat in patterns.items():
print(f"\nheader {header}: {pat}")
for label, vals in cases.items():
hits = [v for v in vals if re.search(pat, v)]
print(f" {label}: {hits}")
PYRepository: ritik4ever/stellar-portfolio-rebalancer
Length of output: 3699
🌐 Web query:
Prometheus blackbox_exporter fail_if_header_not_matches MatchString regex source
💡 Result:
In the Prometheus blackbox_exporter, fail_if_header_not_matches is a configuration block used to fail an HTTP probe if a specific response header does not match a provided regular expression [1][2]. Technical Details and Usage: The fail_if_header_not_matches directive operates on the http_header_match_spec structure [1][3]. This structure requires you to specify a header name, a regular expression, and an optional boolean flag to handle missing headers [4][3]. Key Configuration Parameters: - header: The name of the HTTP header to inspect [4][3]. - regexp: The regular expression string to validate the header value against [4][3]. The blackbox_exporter uses Go's standard library regexp package [4]. - allow_missing: A boolean (defaults to false). If true, the probe will not fail if the header is completely absent from the response [4][3]. If false, the probe will fail if the header is missing [5]. Logic for Multiple Values: When an HTTP response contains multiple values for the same header, fail_if_header_not_matches will succeed (i.e., pass the check) if at least one of the header values matches the provided regular expression [5]. If none of the values match, the probe fails [5]. Example Configuration: The following example from the official documentation demonstrates how to ensure a header matches a specific pattern [2][6]: fail_if_header_not_matches: - header: Access-Control-Allow-Origin regexp: '(*|example.com)' Source Code Implementation: The logic for this check is implemented in the prober/http.go file within the blackbox_exporter repository [5]. It iterates through the values associated with the specified header and uses Regexp.MatchString(val) to perform the validation [5]. If no header values are found and allow_missing is set to false, the probe logs an error and fails [5].
Citations:
- 1: https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
- 2: https://github.com/prometheus/blackbox_exporter/blob/master/example.yml
- 3: https://github.com/prometheus/blackbox_exporter/blob/bee8e9102a106bff63281ee9c64c7b1275ef21d0/CONFIGURATION.md
- 4: https://github.com/prometheus/blackbox_exporter/blob/master/config/config.go
- 5: https://github.com/prometheus/blackbox_exporter/blob/master/prober/http.go
- 6: https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/example.yml
Anchor the response header checks and enforce Connection: Upgrade.
Blackbox Exporter uses Regexp.MatchString, so the current unanchored patterns pass false positives like not-websocket or an embedded accept digest. Also add a Connection: Upgrade check as required by the RFC 6455 handshake.
Proposed fix
fail_if_header_not_matches:
- header: Upgrade
- regexp: (?i)websocket
+ regexp: (?i)^websocket$
+ - header: Connection
+ regexp: (?i)(^|,\s*)Upgrade(\s*,|$)
- header: Sec-WebSocket-Accept
- regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
+ regexp: ^HSmrc0sMlYUkAGmm5OPpG2HaGWk=$📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| fail_if_header_not_matches: | |
| - header: Upgrade | |
| regexp: (?i)websocket | |
| - header: Sec-WebSocket-Accept | |
| regexp: HSmrc0sMlYUkAGmm5OPpG2HaGWk= | |
| fail_if_header_not_matches: | |
| - header: Upgrade | |
| regexp: (?i)^websocket$ | |
| - header: Connection | |
| regexp: (?i)(^|,\s*)Upgrade(\s*,|$) | |
| - header: Sec-WebSocket-Accept | |
| regexp: ^HSmrc0sMlYUkAGmm5OPpG2HaGWk=$ |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/blackbox/blackbox.yml` around lines 25 - 29, Update
the fail_if_header_not_matches checks in the blackbox probe to anchor both
existing regex patterns so they match exactly “websocket” and the expected
Sec-WebSocket-Accept digest, then add a Connection header check requiring the
exact value “Upgrade”.
Source: MCP tools
| annotations: | ||
| summary: Backend WebSocket handshake is failing | ||
| description: The external WebSocket probe is unable to complete a handshake to the backend. | ||
| description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the alert impact statement match the probe’s scope.
A handshake probe does not exercise post-upgrade frame exchange or delivery to already-connected clients. Saying that portfolio and risk pushes “are not reaching connected clients” can mislead responders; describe the observable failure as new WebSocket connections being unable to complete the handshake instead. (rfc-editor.org)
- description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing.
+ description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend, so new WebSocket connections may fail. Inhibited when BackendDown is already firing.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend. Real-time portfolio and risk pushes are not reaching connected clients. Inhibited when BackendDown is already firing. | |
| description: The synthetic WebSocket probe cannot complete an RFC 6455 opening handshake against the backend, so new WebSocket connections may fail. Inhibited when BackendDown is already firing. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deployment/observability/prometheus/alerts.yml` at line 67, Update the
description for the synthetic WebSocket probe alert to state that new WebSocket
connections cannot complete the RFC 6455 opening handshake, rather than claiming
pushes are not reaching already-connected clients. Preserve the existing backend
context and BackendDown inhibition statement.
Source: MCP tools
| 2. echoes an `Upgrade: websocket` response header, and | ||
| 3. returns the `Sec-WebSocket-Accept` digest derived from the probe's key. | ||
|
|
||
| Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Scope the accept-digest claim to the peer that answered.
RFC 6455 permits reverse proxies and load balancers to offload WebSocket management. A correct Sec-WebSocket-Accept proves that the responding peer processed the key; it does not prove that the backend handled the handshake or that application frames flow end to end. (rfc-editor.org)
-Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line.
+Checking the accept digest matters because a `101` alone only proves that the responding peer agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves that peer processed the probe's key; it does not by itself validate the backend or post-handshake application traffic.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Checking the accept digest matters because a `101` alone only proves something in front of the backend agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves the peer that answered is a real WebSocket server that read the probe's key — not a proxy or load balancer echoing a status line. | |
| Checking the accept digest matters because a `101` alone only proves that the responding peer agreed to switch protocols. The digest is `SHA1(key + RFC 6455 GUID)` base64-encoded, so a correct value proves that peer processed the probe's key; it does not by itself validate the backend or post-handshake application traffic. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/OBSERVABILITY.md` at line 34, Revise the accept-digest explanation in
the WebSocket observability documentation to state only that a correct
Sec-WebSocket-Accept proves the peer responding to the probe processed its key.
Remove or qualify claims that it proves the backend handled the handshake or
that application frames flow end to end, while retaining the distinction from a
response based solely on status 101.
Source: MCP tools
2 participants
…cket probe
Alertmanager previously named a receiver
pagerduty-criticalbut pointed it at a local webhook, so nothing actually paged. Replace it with real integrations and make the WebSocket endpoint's availability directly measured rather than inferred.On-call paging (#1492):
oncall-pagerduty(Events API v2) andoncall-opsgeniereceivers, with credentials read from files mounted at /etc/alertmanager/secrets so keys stay out of version control. Route critical alerts to PagerDuty; switching vendors is a one-line receiver change.Synthetic WebSocket probe (#1494):
Upgrade: websocketand returns the Sec-WebSocket-Accept digest derived from the probe key, so a bare 101 from a proxy no longer counts as success.subsystem: api-gatewayso it groups and inhibits alongside the other api-gateway alerts, and add WebSocketProbeStalled to catch an absent probe — otherwise a dead exporter looks healthy.Document the escalation policy, dedup mechanisms, and probe behaviour in docs/OBSERVABILITY.md.
Closes #1492
Closes #1494
Commit Type
Summary
Verification
Release Notes
Summary by CodeRabbit
New Features
Bug Fixes
Documentation