docs: on-call SLO and alerting runbook for core gauges

[onyemaechiezekiel9]

docs: on-call SLO and alerting runbook for core gauges

Closes #753

---

Summary

Authors docs/runbooks/on-call-slo.md — a complete on-call SLO and alerting runbook tying each Prometheus gauge exposed by src/routes/metrics.ts to a threshold, a Prometheus alert rule, and a numbered remediation procedure. Metrics without an interpretation guide are not actionable at 3am; this runbook fixes that.

No source code was modified.

---

Files

docs/runbooks/on-call-slo.md — created
docs/operations-metrics.md — cross-link added in See Also section

---

Gauge Coverage

All gauge names sourced directly from src/routes/metrics.ts — none invented:

Gauge	SLO
disciplr_job_queue_depth	Warning > 100 for 5m · Critical > 500 for 2m
disciplr_db_available_connections	Warning > 80% utilization for 5m · Critical > 95% for 1m
disciplr_db_waiting_clients	Warning > 80% utilization for 5m · Critical > 95% for 1m
disciplr_horizon_listener_lag	Warning > 30 ledgers for 5m · Critical > 120 ledgers for 2m
disciplr_outbox_relay_lag_seconds	Warning/Critical thresholds documented
disciplr_job_failed_total	Rate-based alert documented

---

Runbook Structure (per SLO)

Each of the 4 SLOs includes:

• What it measures — context and business impact
• SLO target — warning and critical thresholds with 7-day rolling window
• Prometheus alert rule — complete YAML ready to paste into alerting config
• Numbered remediation steps — 5–6 actionable steps in priority order
• Escalation policy — timelines and contact matrix
• Silence guidance — when silencing is acceptable and hard limits

Bonus sections: common incident scenarios, full alert config example, and maintenance window guidelines.

---

How to use

# Check a gauge live during an incident
curl -s http://localhost:<PORT>/metrics | grep disciplr_job_queue_depth

The runbook links directly to remediation steps for each alert so on-call engineers can jump straight to the relevant section from a PagerDuty/Alertmanager notification.

---

Security Notes

N/A — documentation only. No credentials, PII, or internal endpoints are referenced in the runbook.

[drips-wave]

@onyemaechiezekiel9 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[1nonlypiece]

having the slo and alerting runbook written down for the core gauges will save on-call a lot of guessing. merged