Add server-side pagination to /api/alerts endpoints

- Add _parse_pagination() helper: page/per_page query params with validation
  (page >= 1, 1 <= per_page <= 500, 400 on invalid or non-integer input)
- _list_alerts() now returns paginated envelope: {alerts, page, per_page,
  total, total_pages}; COUNT(*) shares the same WHERE clause so total
  reflects the filtered set
- schema.sql: add idx_alerts_created_at index on alerts(created_at DESC)
- dashboard.js: always passes per_page=500 so the full queue is visible;
  unwraps data.alerts from the envelope in all three loadAlerts() branches
- Update existing tests (test_app.py, test_rbac.py) to unwrap ["alerts"]
  from the paginated response
- Add 26 new tests in test_pagination.py covering response shape, mechanics,
  filter interaction, empty table, and validation for both endpoints
- README: update test counts to 142 pytest + 87 Go = 229 total

Author: Romil2112

README.md

@@ -46,7 +46,7 @@ Security sits on a few specific choices. The ingest endpoint checks its API key
 - **pgvector semantic similarity** — `POST /api/alerts` stores a fastembed embedding; `GET /api/alerts/<id>/similar` returns the top 5 by cosine distance
 - **Kubernetes manifests** — `deploy/k8s/` covers Deployment, HPA, Ingress, and Namespace with a separate `Dockerfile.ingest-service` for the Go binary
 - **GCP deployment** — `deploy/gcp/` (Cloud Run service YAML + Cloud Build pipeline) and `terraform/gcp/` provision the full stack
-- 116 pytest + 87 Go = **203 tests** covering the ingest API, auth/CSRF, RBAC, KPI math, encryption, audit trail, Kafka consumer, Redis SSE, pgvector similarity, and the full Go ingest handler and gRPC interceptor surface
+- 142 pytest + 87 Go = **229 tests** covering the ingest API, auth/CSRF, RBAC, KPI math, encryption, audit trail, Kafka consumer, Redis SSE, pgvector similarity, pagination, and the full Go ingest handler and gRPC interceptor surface
 
 ## Running the Project
 
@@ -168,7 +168,7 @@ flowchart LR
 
 ## Tests
 
-**116 pytest + 87 Go = 203 tests.** The Python suite covers the ingest API, auth/CSRF, RBAC roles, the classify/escalate flow, KPI math (MTTR, SLA, escalation), filter query params, Fernet encryption at rest, audit trail, SSE live updates, Kafka consumer, Redis pub/sub, pgvector semantic similarity, and the seed and user-management CLIs. The Go suite tests the REST and gRPC ingest handlers, `apiKeyInterceptor` boundary conditions (empty key, no metadata, constant-time comparison, whitespace trimming), W3C traceparent parsing edge cases (Python OTel ≥ 1.44 `flags=03`, zero IDs, extra segments, invalid hex), and proto field mapping (`SourceIp→SourceIP`, `WorkflowRunId→WorkflowRunID`). Both suites run against a real PostgreSQL database (Docker on port 5433 for CI). Point `DATABASE_URL` at a throwaway database and run:
+**142 pytest + 87 Go = 229 tests.** The Python suite covers the ingest API, auth/CSRF, RBAC roles, the classify/escalate flow, KPI math (MTTR, SLA, escalation), filter query params, server-side pagination, Fernet encryption at rest, audit trail, SSE live updates, Kafka consumer, Redis pub/sub, pgvector semantic similarity, and the seed and user-management CLIs. The Go suite tests the REST and gRPC ingest handlers, `apiKeyInterceptor` boundary conditions (empty key, no metadata, constant-time comparison, whitespace trimming), W3C traceparent parsing edge cases (Python OTel ≥ 1.44 `flags=03`, zero IDs, extra segments, invalid hex), and proto field mapping (`SourceIp→SourceIP`, `WorkflowRunId→WorkflowRunID`). Both suites run against a real PostgreSQL database (Docker on port 5433 for CI). Point `DATABASE_URL` at a throwaway database and run:
 
 ```bash
 python -m pytest tests/ -v
@@ -184,7 +184,7 @@ python -m pytest tests/ -v
 | Semantic search | fastembed embeddings stored in pgvector; `GET /api/alerts/<id>/similar` returns top-5 by cosine distance |
 | Horizontal scaling | Redis pub/sub for multi-worker SSE; K8s HPA manifest scales ingest replicas on CPU/RPS |
 | Cloud deployment | Cloud Run + Cloud Build (`deploy/gcp/`); Terraform provisions GCP infra (`terraform/gcp/`) |
-| Test engineering | 203 tests (116 Python + 87 Go) exercising boundary conditions, constant-time comparisons, W3C traceparent edge cases, and proto field mapping without a live database |
+| Test engineering | 229 tests (142 Python + 87 Go) exercising boundary conditions, constant-time comparisons, W3C traceparent edge cases, and proto field mapping without a live database |
 
 ## Roles & Permissions
 

app.py

@@ -437,36 +437,79 @@ def analyst():
 # --------------------------------------------------------------------------- #
 # API
 # --------------------------------------------------------------------------- #
-def _list_alerts(where_sql, params):
-    """Run the shared alert-list query and return decrypted, serialized rows."""
+def _parse_pagination():
+    """Parse and validate page/per_page from the request query string.
+
+    Defaults: page=1, per_page=100. Aborts 400 on non-integer or out-of-range values.
+    per_page is capped at 500 to prevent unbounded result sets.
+    """
+    try:
+        page = int(request.args.get("page", 1))
+    except (ValueError, TypeError):
+        abort(400, description="page must be a positive integer")
+    try:
+        per_page = int(request.args.get("per_page", 100))
+    except (ValueError, TypeError):
+        abort(400, description="per_page must be an integer between 1 and 500")
+    if page < 1:
+        abort(400, description="page must be >= 1")
+    if per_page < 1 or per_page > 500:
+        abort(400, description="per_page must be between 1 and 500")
+    return page, per_page
+
+
+def _list_alerts(where_sql, params, page: int, per_page: int) -> dict:
+    """Run the shared alert-list query and return a paginated envelope.
+
+    The COUNT uses the same WHERE clause so ``total`` reflects the filtered
+    set, not the whole table.
+    """
+    offset = (page - 1) * per_page
     with get_conn() as conn, conn.cursor() as cur:
         # nosec B608: where_sql is assembled only from the whitelisted
         # FILTER_COLUMNS names; every value is bound as a parameter.
-        cur.execute("SELECT * FROM alerts" + where_sql + _SEVERITY_ORDER, params)  # nosec B608
+        cur.execute("SELECT count(*) AS c FROM alerts" + where_sql, params)  # nosec B608
+        total = cur.fetchone()["c"]
+        cur.execute(  # nosec B608
+            "SELECT * FROM alerts" + where_sql + _SEVERITY_ORDER + " LIMIT %s OFFSET %s",
+            params + [per_page, offset],
+        )
         rows = cur.fetchall()
-    return [serialize(decrypt_alert(r)) for r in rows]
+    alerts = [serialize(decrypt_alert(r)) for r in rows]
+    total_pages = (total + per_page - 1) // per_page if total else 0
+    return {
+        "alerts": alerts,
+        "page": page,
+        "per_page": per_page,
+        "total": total,
+        "total_pages": total_pages,
+    }
 
 
 @app.route("/api/alerts")
 @login_required
 def api_open_alerts():
     """Open queue, optionally filtered by severity/source/assignee.
 
-    CRITICAL -> LOW then newest first.
+    CRITICAL -> LOW then newest first. Paginated: page/per_page params
+    (default page=1, per_page=100, max per_page=500).
     """
+    page, per_page = _parse_pagination()
     where_sql, params = alert_filters(extra=[("status = %s", "open")])
-    return jsonify(_list_alerts(where_sql, params))
+    return jsonify(_list_alerts(where_sql, params, page, per_page))
 
 
 @app.route("/api/alerts/all")
 @login_required
 def api_all_alerts():
     """Every alert, optionally filtered by severity/source/assignee.
 
-    CRITICAL -> LOW then newest first.
+    CRITICAL -> LOW then newest first. Paginated: page/per_page params
+    (default page=1, per_page=100, max per_page=500).
     """
+    page, per_page = _parse_pagination()
     where_sql, params = alert_filters()
-    return jsonify(_list_alerts(where_sql, params))
+    return jsonify(_list_alerts(where_sql, params, page, per_page))
 
 
 def _valid_api_key():

schema.sql

@@ -57,11 +57,12 @@ CREATE TABLE analyst_actions (
     response_time_seconds INT
 );
 
-CREATE INDEX idx_alerts_status   ON alerts(status);
-CREATE INDEX idx_alerts_severity ON alerts(severity);
-CREATE INDEX idx_alerts_category ON alerts(category);
-CREATE INDEX idx_alerts_source   ON alerts(source);
-CREATE INDEX idx_alerts_assigned ON alerts(assigned_to);
+CREATE INDEX idx_alerts_status     ON alerts(status);
+CREATE INDEX idx_alerts_severity   ON alerts(severity);
+CREATE INDEX idx_alerts_category   ON alerts(category);
+CREATE INDEX idx_alerts_source     ON alerts(source);
+CREATE INDEX idx_alerts_assigned   ON alerts(assigned_to);
+CREATE INDEX idx_alerts_created_at ON alerts(created_at DESC);
 CREATE INDEX idx_actions_alert   ON analyst_actions(alert_id);
 CREATE INDEX idx_actions_analyst ON analyst_actions(analyst_name);
 

static/dashboard.js

@@ -49,8 +49,8 @@ function filterQuery() {
   if (f.severity) qs.set("severity", f.severity);
   if (f.source) qs.set("source", f.source);
   if (f.assigned_to) qs.set("assigned_to", f.assigned_to);
-  const s = qs.toString();
-  return s ? `?${s}` : "";
+  qs.set("per_page", "500");
+  return `?${qs.toString()}`;
 }
 
 // Returns query string for the active preset (merged with filter selects).
@@ -72,8 +72,8 @@ function presetQuery() {
     qs.set("created_after", today.toISOString());
   }
   // "escalated" and "all-open" handled in loadAlerts
-  const s = qs.toString();
-  return s ? `?${s}` : "";
+  qs.set("per_page", "500");
+  return `?${qs.toString()}`;
 }
 
 async function applyPreset(preset) {
@@ -156,14 +156,16 @@ async function loadAlerts() {
   let alerts;
   if (_activePreset === "escalated") {
     const res = await fetch("/api/alerts/all" + filterQuery());
-    const all = await res.json();
-    alerts = all.filter(a => a.status === "escalated");
+    const data = await res.json();
+    alerts = (data.alerts || []).filter(a => a.status === "escalated");
   } else if (_activePreset === "all-open") {
     const res = await fetch("/api/alerts" + filterQuery());
-    alerts = await res.json();
+    const data = await res.json();
+    alerts = data.alerts || [];
   } else {
     const res = await fetch("/api/alerts" + presetQuery());
-    alerts = await res.json();
+    const data = await res.json();
+    alerts = data.alerts || [];
   }
   const tbody = document.getElementById("alert-rows");
   if (!tbody) return;

tests/test_app.py

@@ -14,13 +14,13 @@ def test_pages_render(client):
 
 
 def test_open_alerts_only_and_severity_sorted(client):
-    rows = client.get("/api/alerts").get_json()
+    rows = client.get("/api/alerts").get_json()["alerts"]
     assert [r["id"] for r in rows] == [4, 3]          # HIGH before LOW; only open
     assert all(r["status"] == "open" for r in rows)
 
 
 def test_all_alerts_returns_everything(client):
-    rows = client.get("/api/alerts/all").get_json()
+    rows = client.get("/api/alerts/all").get_json()["alerts"]
     assert len(rows) == 4
 
 
@@ -56,23 +56,23 @@ def test_stats_by_source_and_assignees(client):
 
 
 def test_filter_alerts_by_severity(client):
-    rows = client.get("/api/alerts/all?severity=CRITICAL").get_json()
+    rows = client.get("/api/alerts/all?severity=CRITICAL").get_json()["alerts"]
     assert sorted(r["id"] for r in rows) == [1, 2]
 
 
 def test_filter_alerts_by_source(client):
-    rows = client.get("/api/alerts/all?source=EDR").get_json()
+    rows = client.get("/api/alerts/all?source=EDR").get_json()["alerts"]
     assert [r["id"] for r in rows] == [2]
 
 
 def test_filter_alerts_by_assignee(client):
-    rows = client.get("/api/alerts/all?assigned_to=bob").get_json()
+    rows = client.get("/api/alerts/all?assigned_to=bob").get_json()["alerts"]
     assert [r["id"] for r in rows] == [2]
 
 
 def test_filter_combines_with_open_queue(client):
     # The open queue honors filters too: only open + HIGH -> alert 4.
-    rows = client.get("/api/alerts?severity=HIGH").get_json()
+    rows = client.get("/api/alerts?severity=HIGH").get_json()["alerts"]
     assert [r["id"] for r in rows] == [4]
 
 
@@ -85,7 +85,7 @@ def test_classify_updates_status_and_closes_alert(client):
     assert resp.status_code == 200
     assert resp.get_json()["status"] == "true_positive"
     # alert 4 should no longer appear in the open queue
-    open_ids = [r["id"] for r in client.get("/api/alerts").get_json()]
+    open_ids = [r["id"] for r in client.get("/api/alerts").get_json()["alerts"]]
     assert 4 not in open_ids
 
 
@@ -166,7 +166,7 @@ def test_ingest_alert_creates_open_alert(client):
     assert created["workflow_run_id"] == "wf-abc123"
     assert "push_ref" in created["run_metadata"]
     # the ingested alert is now in the open queue
-    open_titles = [r["title"] for r in client.get("/api/alerts").get_json()]
+    open_titles = [r["title"] for r in client.get("/api/alerts").get_json()["alerts"]]
     assert payload["title"] in open_titles
 
 
@@ -248,7 +248,7 @@ def test_retention_purges_only_old_alerts(client):
     # Fixture alert 3 is 2 days old; the rest are <= 1 hour old.
     deleted = soc_app.purge_old_alerts(1)
     assert deleted == 1
-    remaining = [r["id"] for r in client.get("/api/alerts/all").get_json()]
+    remaining = [r["id"] for r in client.get("/api/alerts/all").get_json()["alerts"]]
     assert 3 not in remaining
     assert sorted(remaining) == [1, 2, 4]
 
@@ -257,7 +257,7 @@ def test_retention_disabled_is_noop(client):
     import app as soc_app
 
     assert soc_app.purge_old_alerts(0) == 0
-    assert len(client.get("/api/alerts/all").get_json()) == 4
+    assert len(client.get("/api/alerts/all").get_json()["alerts"]) == 4
 
 
 def test_sensitive_fields_encrypted_at_rest(client, monkeypatch):