Application metrics and Sysdig dashboards

Summary for nr-site-registry: instrument the backend so each API/GraphQL operation reports success vs failure, expose Prometheus-format metrics, and visualize them in Sysdig Monitor.

Goals

See per-operation health: successes, failures, and volume over time.
Use Sysdig for dashboards (org standard on BC Gov OpenShift).
Start in test; extend to prod after validation.
Out of scope for v1: alerts, ticket automation, full SLO program.

How the pieces fit together

Layer	What it does
Application	Increments counters on each GraphQL operation (success/failure).
`/metrics` endpoint	Exposes counters in Prometheus text format.
Platform scrape	Sysdig/Prometheus pulls `/metrics` from backend pods (often via pod annotations).
Sysdig dashboards	PromQL panels: rates, totals, breakdown by `operation`.

Frontend → GraphQL → NestJS backend
                        ↓ (plugin records outcome)
                   GET /metrics
                        ↓ (scrape every ~15–60s)
                   Sysdig Monitor → Dashboards

SysdigTeam CR (in <license-plate>-tools) grants access to Sysdig. It does not create application metrics. App metrics require code + scrape + dashboards.

Prerequisites

Sysdig team exists and is Ready in <license-plate>-tools (e.g. c6a6e5-tools, e38158-tools).
Teammates use the same email in the CR as their SSO login to Sysdig.
Access to test app namespace (e.g. c6a6e5-test) for deploy and dashboard filters.

Useful commands (tools namespace)

oc api-resources | grep -i sysdig
oc get sysdig-teams -n <license-plate>-tools
oc get sysdig-team <name> -n <license-plate>-tools -o yaml

Resource name on cluster is usually sysdig-teams (not sysdigteam). Kind in YAML: SysdigTeam, apiVersion: ops.gov.bc.ca/v1alpha1.

Part 1 — Instrument the backend (code)

Stack: NestJS 10, GraphQL (Apollo Federation), backend/.

1.1 Add dependency

cd backend && npm install prom-client

1.2 Metrics to expose

Five metric families (prefix site_registry_):

Metric	Type	Labels	Purpose
`site_registry_graphql_operations_total`	Counter	`operation`, `outcome`, `error_class`	Success/failure per GraphQL operation
`site_registry_graphql_operation_duration_seconds`	Histogram	`operation`, `outcome`	GraphQL latency per operation
`site_registry_http_requests_total`	Counter	`method`, `route`, `status`, `outcome`, `error_class`	HTTP requests (excludes `/metrics`)
`site_registry_http_request_duration_seconds`	Histogram	`method`, `route`, `outcome`	HTTP request latency
`site_registry_auth_failures_total`	Counter	`reason`, `guard`	Wire 401/403 only

GraphQL labels:

operation: GraphQL operationName (e.g. searchSites), or anonymous if missing.
outcome: success or failure (GraphQL errors, success: false, or payload httpStatusCode ≥ 400).
error_class: na on success; client, server, or unknown on failure.

Auth labels: reason = unauthorized (401) or forbidden (403); guard = http today.

Avoid high-cardinality labels (user IDs, raw IDs in paths).

1.3 Suggested files

File	Role
`backend/src/app/metrics/operational-metrics.service.ts`	Defines counters; `recordGraphql(operation, outcome, errorClass)`
`backend/src/app/metrics/graphql-metrics.plugin.ts`	Apollo `@Plugin()` — records on `willSendResponse`
`backend/src/app/metrics/metrics.module.ts`	Providers + exports
`backend/src/main.ts`	Register `GET /metrics`
`backend/src/app.module.ts`	Import `MetricsModule`

1.4 GraphQL plugin (why not every resolver?)

GraphQL can return HTTP 200 with "errors": [...] — ingress HTTP metrics look healthy while the operation failed.
One Apollo plugin covers all operations consistently.
No changes needed in individual resolver files for basic success/failure.

Hook: requestDidStart → willSendResponse → read operationName, check response.body.singleResult.errors, increment counter.

1.5 Expose `/metrics`

After NestFactory.create, register on the Express instance:

http.get('/metrics', async (_req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Auth: Keycloak guards must not block the scraper. Use @Unprotected() on a metrics route or exclude /metrics from guards.

1.6 Local verification

npm run start:dev
curl http://localhost:4007/metrics | grep site_registry
# Run GraphQL operations, then curl again — counters should increase

Part 2 — Enable scrape in Kubernetes (test)

Add pod annotations on the backend deployment (charts/app/templates/backend/templates/deployment.yaml):

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: {{ .Values.backend.service.targetPort | quote }}
    prometheus.io/path: "/metrics"

The chart sets PORT and containerPort from the same backend.service.targetPort. Confirm the live port on the pod (test has used 3000; local dev often 4007; charts/app/values.yaml may list another value per environment).

Deploy to test via the normal pipeline.

2.1 Find the backend pod (test example)

Helm release in test is typically nr-site-registry-test. Backend resources use the backend component name.

Resource	Example name (test)
Namespace	`c6a6e5-test` (replace `<license-plate>-test`)
Deployment	`nr-site-registry-test-backend`
Pod	`nr-site-registry-test-backend-<replicaset-hash>-<id>`
Labels	`app.kubernetes.io/name=backend`, `app.kubernetes.io/instance=nr-site-registry-test`

oc project c6a6e5-test

# List backend pods (pick one in Running, READY 1/1)
oc get pods -l app.kubernetes.io/name=backend,app.kubernetes.io/instance=nr-site-registry-test

# Example pod name (yours will differ after redeploy):
# nr-site-registry-test-backend-7f4dc577df-7fkhf

2.2 Confirm scrape annotations and listen port

Before port-forward, read PORT and prometheus.io/port from the running pod (do not assume 8080):

POD=nr-site-registry-test-backend-7f4dc577df-7fkhf   # replace with your pod name

oc describe pod "$POD" | grep -E 'prometheus.io|^\s+PORT:|containerPort|Liveness|Readiness'

Expected on a metrics-enabled test deploy:

prometheus.io/scrape: true
prometheus.io/path: /metrics
prometheus.io/port: 3000 (must match PORT env and container port)
PORT: 3000, probes on :3000

2.3 Port-forward and curl `/metrics` (two terminals)

Use the container port from the previous step as the remote port (3000 in test). Pick any free local port (e.g. 13000) to avoid clashing with other apps on 8080/3000:

Terminal 1 (leave running):

oc project c6a6e5-test
oc port-forward pod/"$POD" -c nr-site-registry-test-backend 13000:3000

You should see Forwarding from 127.0.0.1:13000 -> 3000. If you see connection refused on 8080, the app is not listening there — use 3000 (or whatever describe shows).

Terminal 2:

# Metric families (may be HELP/TYPE only until traffic exists)
curl -s http://127.0.0.1:13000/metrics | grep -E '^# (HELP|TYPE) site_registry'

# Full check with headers
curl -v http://127.0.0.1:13000/metrics | head -30

Pass: HTTP/1.1 200 OK, Content-Type: text/plain, and five site_registry_* HELP lines.

2.4 Generate UI traffic, then verify counters

Use test Site Registry in the browser (search, open a site, logged in if possible).
Re-run:

curl -s http://127.0.0.1:13000/metrics | grep site_registry_graphql_operations_total

# Sample lines only (no comments)
curl -s http://127.0.0.1:13000/metrics | grep -v '^#' | grep site_registry

Pass: Named operations appear, e.g. operation="searchSites",outcome="success". Low or no operation="anonymous" after real UI use.

Note: Many outcome="failure",error_class="client" lines are common when the API returns success: false (permissions, not logged in, empty cart). That is application classification, not broken metrics. Use Sysdig rate() and filter error_class for server/unknown when judging system health.

2.5 Verify inside the pod (optional, no port-forward)

oc exec "$POD" -c nr-site-registry-test-backend -- \
  sh -c 'wget -qO- http://127.0.0.1:3000/metrics 2>&1 | head -20'

If this works but local port-forward fails, fix local port mapping or a stale pod name.

2.6 Port-forward via Service (alternative)

oc get svc -n c6a6e5-test | grep backend
oc port-forward -n c6a6e5-test svc/nr-site-registry-test-backend 13000:3000
curl -s http://127.0.0.1:13000/metrics | grep site_registry

If metrics work on the pod but not in Sysdig after ~30–60 minutes, ask the Teams channel OpenShift-howto-sysdig whether additional scrape config (e.g. ServiceMonitor) is required.

Part 3 — Sysdig: confirm metrics, then build dashboards

Prerequisite: Part 2 pod verification succeeds (curl → 200, counters after UI traffic).

3.1 Explore (metric exists?)

Log in to Sysdig Monitor (platform SSO).
Open Explore / Metrics (Prometheus).
Search: site_registry_graphql_operations_total.
Filter by namespace c6a6e5-test (or your <license-plate>-test), workload nr-site-registry-test-backend, cluster.

If the metric does not appear, fix scrape/code before building dashboards.

3.1.1 Prove Explore matches pod metrics

Note the time; run search + open a site in test UI.
In Explore, run:

sum(rate(site_registry_graphql_operations_total{kube_namespace_name="c6a6e5-test"}[5m])) by (operation)

Pass: Non-zero rates for searchSites, findSiteBySiteId, or findSiteBySiteIdLoggedInUser around that time.

3.2 Example PromQL panels

Replace namespace with yours (e.g. c6a6e5-test):

Failures (24h) — stat panel

sum(increase(site_registry_graphql_operations_total{outcome="failure", kube_namespace_name="c6a6e5-test"}[24h]))

Successes (24h) — stat panel

sum(increase(site_registry_graphql_operations_total{outcome="success", kube_namespace_name="c6a6e5-test"}[24h]))

Failure rate by operation — time series

sum(rate(site_registry_graphql_operations_total{outcome="failure", kube_namespace_name="c6a6e5-test"}[5m])) by (operation)

Success rate by operation — time series

sum(rate(site_registry_graphql_operations_total{outcome="success", kube_namespace_name="c6a6e5-test"}[5m])) by (operation)

Error percentage by operation

sum(rate(site_registry_graphql_operations_total{outcome="failure", kube_namespace_name="c6a6e5-test"}[5m])) by (operation)
/
sum(rate(site_registry_graphql_operations_total{kube_namespace_name="c6a6e5-test"}[5m])) by (operation)

Use Explore to confirm exact label names (kube_namespace_name, kube_workload_name, etc.) — they can vary slightly by platform.

3.3 Dashboard: SiteRegistry-Test-Graphql (built)

Full panel catalog with titles, types, PromQL, and how to read each panel: Operational metrics dashboard — panel reference

Quick summary

Section	Panels	Metric families
OVERVIEW	1	(text)
GRAPHQL HEALTH	8	`graphql_operations_total`, `graphql_operation_duration_seconds`
HTTP LAYER	5	`http_requests_total`, `http_request_duration_seconds`
AUTH	2	`auth_failures_total`

Incident panel: Enquiry failures with error_class=~"server|unknown".

Dashboard scope: namespace c6a6e5-test, workload nr-site-registry-test-backend.
Default time range: Last 6 hours for rate[5m] charts; Last 24 hours for Number panels using increase[6h] / [24h].

Part 4 — Impact on the existing app

Area	Expected impact
GraphQL API / business logic	No change to contracts or behavior
Performance	Negligible (counter increment per request)
Risk	Plugin must not throw; `/metrics` must not be blocked by auth
Deploy	Normal backend release to test, then prod

Rollback: redeploy previous image; remove annotations if needed.

Definition of done

oc describe pod shows prometheus.io/scrape, prometheus.io/path=/metrics, and prometheus.io/port matching PORT (e.g. 3000 in c6a6e5-test).
oc port-forward to that port (e.g. 13000:3000) and curl returns 200 with all site_registry_* HELP lines.
After UI traffic, curl shows site_registry_graphql_operations_total samples (e.g. searchSites, findSiteBySiteIdLoggedInUser).
Sysdig Explore finds the metric with kube_namespace_name="c6a6e5-test" (or your test namespace).
Dashboard SiteRegistry-Test-Graphql with 4 sections and 16 panels (test).
Doc updated with metric names and panel reference.
Clone dashboard for prod (change namespace/workload only).
user_audience label for external-user monitoring at go-live.

Troubleshooting

Symptom	Likely cause
`port-forward` → `connection refused` on 8080 inside pod	Backend listens on another port (test: 3000). Run `oc describe pod` and forward `LOCAL:3000`.
`curl: (52) Empty reply from server`	Port-forward not running, wrong local port, or tunnel lost after refused backend port
`/metrics` returns 401	Keycloak blocking scraper
`/metrics` 404	Route not registered
Only HELP/TYPE, no sample lines	No traffic yet; use UI then curl again
Local counters OK, Sysdig empty	Missing/wrong scrape annotations or platform config; confirm `prometheus.io/port` matches listen port
Only `anonymous` operation	Clients not sending `operationName`
Many `failure` / `client` on site-detail ops	Often expected (`success: false` in JSON when not authorized or missing data)
Teammate can't see Sysdig team	Email in CR ≠ SSO login email; or CR not Reconciled

Platform help: Microsoft Teams OpenShift-howto-sysdig

Work split (ticket-friendly)

Task	Owner
Backend metrics + `/metrics`	App team
Helm scrape annotations + test deploy	App / DevOps
Confirm scrape in Sysdig	Platform / Teams OpenShift-howto-sysdig if stuck
Dashboard panels	App team
SysdigTeam / access	Already in `<license-plate>-tools` (edit users via CR only)

References

Sysdig Monitor – team setup
RED metrics: rate, errors, duration
Operational metrics overview

Navigation Menu

Uh oh!

Application metrics and Sysdig dashboards

Goals

How the pieces fit together

Prerequisites

Useful commands (tools namespace)

Part 1 — Instrument the backend (code)

1.1 Add dependency

1.2 Metrics to expose

1.3 Suggested files

1.4 GraphQL plugin (why not every resolver?)

1.5 Expose `/metrics`

1.6 Local verification

Part 2 — Enable scrape in Kubernetes (test)

2.1 Find the backend pod (test example)

2.2 Confirm scrape annotations and listen port

2.3 Port-forward and curl `/metrics` (two terminals)

2.4 Generate UI traffic, then verify counters

2.5 Verify inside the pod (optional, no port-forward)

2.6 Port-forward via Service (alternative)

Part 3 — Sysdig: confirm metrics, then build dashboards

3.1 Explore (metric exists?)

3.1.1 Prove Explore matches pod metrics

3.2 Example PromQL panels

3.3 Dashboard: SiteRegistry-Test-Graphql (built)

Part 4 — Impact on the existing app

Definition of done

Troubleshooting

Work split (ticket-friendly)

References

Uh oh!

FilesExpand file tree

application-metrics-sysdig.md

Latest commit

History

application-metrics-sysdig.md

File metadata and controls

Application metrics and Sysdig dashboards

Goals

How the pieces fit together

Prerequisites

Useful commands (tools namespace)

Part 1 — Instrument the backend (code)

1.1 Add dependency

1.2 Metrics to expose

1.3 Suggested files

1.4 GraphQL plugin (why not every resolver?)

1.5 Expose /metrics

1.6 Local verification

Part 2 — Enable scrape in Kubernetes (test)

2.1 Find the backend pod (test example)

2.2 Confirm scrape annotations and listen port

2.3 Port-forward and curl /metrics (two terminals)

2.4 Generate UI traffic, then verify counters

2.5 Verify inside the pod (optional, no port-forward)

2.6 Port-forward via Service (alternative)

Part 3 — Sysdig: confirm metrics, then build dashboards

3.1 Explore (metric exists?)

3.1.1 Prove Explore matches pod metrics

3.2 Example PromQL panels

3.3 Dashboard: SiteRegistry-Test-Graphql (built)

Part 4 — Impact on the existing app

Definition of done

Troubleshooting

Work split (ticket-friendly)

References

1.5 Expose `/metrics`

2.3 Port-forward and curl `/metrics` (two terminals)