Metrics dashboard improvements (#458)

* Add diagnostic logging when pipeline receives data without cloud exporter

handleMetrics(), handleSpans(), and handleLogs() previously returned
silently when the cloud exporter was nil, causing complete data loss with
no diagnostic signal. Add a sync.Once warning on first invocation so
operators can see that telemetry data is being received but dropped.

* Resolve GCP project ID from metadata server as fallback

LoadConfig() now queries the GCE metadata server when no explicit
SCION_GCP_PROJECT_ID or credentials file provides a project ID.
The compute/metadata package respects GCE_METADATA_HOST, so this
transparently reaches the scion metadata emulator (localhost:18380)
in agent containers. A 2-second timeout prevents blocking startup
when no metadata server is reachable.

This fixes the primary blocker where the pipeline cloud exporter
failed with "GCP project ID is required" on instances like
scion-integration2 that have a working metadata server but no
explicit credentials file or env var.

* Route hook metrics through pipeline to prevent sampling rate violations

Each sciontool hook invocation previously created its own Cloud Monitoring
exporter, causing ~50% of metric writes to be rejected when hooks fired
within Cloud Monitoring's minimum sampling period.

Two changes fix this:
1. In GCP mode, short-lived processes (hooks, batch=false) now send
   metrics via OTLP to the local pipeline receiver instead of directly
   to Cloud Monitoring. This mirrors how logs already work.
2. The pipeline buffers incoming metrics and flushes to Cloud Monitoring
   every 15 seconds, consolidating rapid writes into safe batches.

Long-lived processes (init, batch=true) continue exporting directly
to Cloud Monitoring since they maintain stable cumulative counters.

* Register ModelResponse hook to enable token metrics from Claude Code

Claude Code's ModelResponse hook event carries per-turn token counts
(input_tokens, output_tokens, cached_tokens) but was not registered
in the embedded settings.json. The dialect parser and metric recording
code already handle this event type — only the registration was missing.

This enables gen_ai.tokens.input, gen_ai.tokens.output, and
gen_ai.tokens.cached metrics to be recorded from each model turn.
Also updates MaxModelCalls capability to SupportYes since model-end
events are now emitted.

* Skip metadata server query in tests to avoid unnecessary network calls

Move the testing.Testing() check before the metadata query instead of
after, so tests never make a network call to the metadata server only
to have the result discarded by the defense-in-depth guard.

* Update capability test to match ModelResponse hook registration

The Claude Code harness now supports MaxModelCalls (SupportYes) since
ModelResponse hooks are registered. Update the test expectation.

* Deduplicate buffered metrics before export to Cloud Monitoring

When multiple hook processes report the same cumulative counter (e.g.
agent.tool.calls with tool=Read) within the 15-second buffer window,
their data points conflict in Cloud Monitoring's sampling rate check.

The pipeline now deduplicates by (metric name, attribute set) before
exporting, keeping only the latest data point per combination. This
eliminates the remaining sampling-rate violations that the buffering
alone didn't solve.

* Remove metadata server fallback for GCP project ID resolution

The GCP project ID for metrics should be derived from the injected
telemetry service account credentials file, not the metadata server.
The metadata server returns the infrastructure project (where the VM
runs), which may differ from the intended metrics target project
specified in the SA key JSON.

* Enforce minimum interval between metric buffer flushes

Track the last successful export time and skip flushes that occur within
metricFlushInterval (15s) of the previous export. This prevents the
shutdown drain in Pipeline.Stop() from racing with a recent periodic
flush, which caused Cloud Monitoring to reject cumulative data points
written less than 10 seconds apart.

* Add metrics dashboard to hub admin UI

Backend: MetricsDashboardService queries Cloud Monitoring for Scion
telemetry metrics (sessions, API calls, tokens, active agents).
Supports summary, sessions, model-calls, and tokens views with
daily aggregation and 5-minute result caching.

Frontend: Lit page component with Chart.js charts, tabs for each
view, and period selector (7/14/30 days). Wired into admin nav
and route system.

* Address code review: fix error handling and label extraction

- Return errors on partial Cloud Monitoring query failures instead of
  silently caching empty results. Only cache fully successful responses.
  Partial data is still returned with an X-Metrics-Warning header.
- Extract the specific requested label key from groupByLabel instead of
  iterating over arbitrary map entries. Prevents mislabeling when series
  contain multiple metric labels.

* Fix ALIGN_SUM → ALIGN_DELTA for cumulative metrics

Cloud Monitoring requires ALIGN_DELTA (not ALIGN_SUM) for CUMULATIVE
counter metrics. ALIGN_SUM is only valid for GAUGE metrics.

* feat: move metrics dashboard from admin to main nav with relaxed auth

Phase 1 of metrics dashboard refactoring:
- Rename admin-metrics.ts → metrics-dashboard.ts, update element tag
  from scion-page-admin-metrics to scion-page-metrics
- Add /metrics route to main ROUTES, keep /admin/metrics as backward-compat redirect
- Move "Metrics" nav item from Admin section to Management section in sidebar
- Relax backend authorization from admin-only to all authenticated users
- Register new /api/v1/metrics/ endpoint, keep legacy /api/v1/admin/metrics-dashboard
- Update app-shell PAGE_TITLES
- Remove scion-page-admin-metrics from ADMIN_ROUTES set

* feat: add per-project metrics views with parameterized queries

Phase 2 of metrics dashboard refactoring:

Backend:
- Add QueryOption/WithProjectID functional options pattern for query parameterization
- Extend all query methods (QuerySummary, QuerySessions, QueryModelCalls, QueryTokens)
  to accept ...QueryOption and thread project_id filter through low-level query functions
- Add extraFilter parameter to queryGroupedTimeSeries, queryUniqueLabels,
  queryDailyUniqueCount for project-scoped Cloud Monitoring queries
- Add handleProjectMetricsDashboard handler with project view authorization
- Register /projects/:id/metrics sub-route in handleProjectRoutes
- Filter on metric.labels.grove_id (canonical label from GCP exporter)

Frontend:
- Add projectId property to metrics-dashboard.ts component
- Parse projectId from URL on connectedCallback (/projects/:id/metrics pattern)
- Route API calls to project-scoped endpoint when projectId is set
- Update header title to show "Project Metrics" when project-scoped
- Add /projects/:id/metrics route in main.ts (before catch-all project route)
- Add "Project Metrics" page title pattern in app-shell.ts
- Add "Metrics" button in project-detail.ts header-actions

* feat: add metrics summary row to project detail page

Phase 3 of metrics dashboard refactoring:

Backend:
- Add ProjectMetricsSummary struct with 24h scalar counters
  (sessions, API calls, tokens, active agents)
- Add QueryProjectSummary method with lightweight scalar-aggregate queries
- Add handleProjectMetricsSummary handler with project view authorization
- Register /projects/:id/metrics-summary sub-route (before /metrics to avoid
  prefix-match collision)
- Return {available: false} when metrics service is not configured

Frontend:
- Add metricsSummary state variable to project-detail.ts
- Fetch summary in loadData() as non-blocking parallel call
- Render metrics stats row below existing stats row when data available
- Include "View Details" link to /projects/:id/metrics
- Gracefully hide metrics row when unavailable (no error states)
- Add formatTokenCount helper for human-readable token numbers

* fix: address code review findings for metrics dashboard

- Fix H-1: Remove dead /dashboard suffix from frontend API path — use
  /api/v1/metrics/ directly to match backend registration
- Fix H-2: Add explicit code comment on handleAdminMetricsDashboard
  documenting the intentional auth relaxation on the legacy endpoint
- Fix M-1: Mark unused subPath parameter with _ in handleProjectMetricsDashboard
- Fix M-2: Use cacheKeySuffix() helper to avoid trailing colon in cache keys
  when ProjectID is empty (global queries)

* Add graph-up icon to Shoelace icon allowlist

The metrics dashboard nav item uses the graph-up Bootstrap Icon but it
was missing from the USED_ICONS array in the copy script, so it was
never copied to the public assets directory.

* Address PR #458 review findings: shutdown race, dedup stability, sort order

Pipeline (pipeline.go):
- Add sync.WaitGroup to coordinate metric flush goroutine shutdown,
  preventing race between final flush and ticker-driven loop
- Add force parameter to flushMetricBuffer to bypass interval check
  during shutdown, preventing loss of buffered metrics
- Sort OTel attributes by key in attrSetKey to ensure stable
  deduplication keys regardless of protobuf attribute ordering

Backend (metrics_dashboard.go):
- Sort queryGroupedTimeSeries results by label for deterministic API
  responses and stable chart legend ordering
- Sort queryDailyUniqueCount results by timestamp for chronological
  ordering

Frontend (metrics-dashboard.ts):
- Move chart.destroy() inside requestAnimationFrame callback to prevent
  overlapping Chart.js instances during rapid tab switching

* Fix CI failures: gofmt formatting and errcheck in metrics dashboard

Run gofmt on metrics_dashboard.go (const block alignment) and
server.go (struct field alignment). Handle the error return from
metricsDashboard.Close() during shutdown to satisfy errcheck.

* Use project_id label instead of grove_id in metrics filter

The compat-literals CI check prohibits new grove terminology. The
telemetry exporter emits both grove_id and project_id labels, so
filtering on project_id is correct and follows project vocabulary.

---------

Co-authored-by: Scion Agent (metrics-validation-lead) <agent@scion.dev>

Author: ptone

go.mod

@@ -5,6 +5,7 @@ go 1.26.1
 require (
 	cloud.google.com/go/compute/metadata v0.9.0
 	cloud.google.com/go/logging v1.13.2
+	cloud.google.com/go/monitoring v1.24.3
 	cloud.google.com/go/secretmanager v1.16.0
 	cloud.google.com/go/storage v1.59.1
 	entgo.io/ent v0.14.5
@@ -71,7 +72,6 @@ require (
 	cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
 	cloud.google.com/go/iam v1.5.3 // indirect
 	cloud.google.com/go/longrunning v0.8.0 // indirect
-	cloud.google.com/go/monitoring v1.24.3 // indirect
 	cloud.google.com/go/trace v1.11.7 // indirect
 	github.com/Azure/go-ntlmssp v0.1.1 // indirect
 	github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect

pkg/harness/capabilities_test.go

@@ -47,7 +47,7 @@ func TestAdvancedCapabilitiesDefaults(t *testing.T) {
 			name:                "claude",
 			harness:             "claude",
 			expectMaxTurns:      api.SupportYes,
-			expectMaxModelCalls: api.SupportNo,
+			expectMaxModelCalls: api.SupportYes,
 			expectMaxDuration:   api.SupportYes,
 			expectAuthFile:      api.SupportYes,
 			expectVertexAI:      api.SupportYes,

pkg/harness/claude/embeds/settings.json

@@ -20,6 +20,7 @@
     "SubagentStop": [{"matcher": "*", "hooks": [{"name": "scion-status", "type": "command", "command": "sciontool hook --dialect=claude"}]}],
     "PreToolUse": [{"matcher": "*", "hooks": [{"name": "scion-status", "type": "command", "command": "sciontool hook --dialect=claude"}]}],
     "PostToolUse": [{"matcher": "*", "hooks": [{"name": "scion-status", "type": "command", "command": "sciontool hook --dialect=claude"}]}],
+    "ModelResponse": [{"matcher": "*", "hooks": [{"name": "scion-telemetry", "type": "command", "command": "sciontool hook --dialect=claude"}]}],
     "Notification": [{"matcher": "ToolPermission", "hooks": [{"name": "scion-status", "type": "command", "command": "sciontool hook --dialect=claude"}]}]
   }
 }

pkg/harness/claude_code.go

@@ -41,7 +41,7 @@ func (c *ClaudeCode) AdvancedCapabilities() api.HarnessAdvancedCapabilities {
 		Harness: "claude",
 		Limits: api.HarnessLimitCapabilities{
 			MaxTurns:      api.CapabilityField{Support: api.SupportYes},
-			MaxModelCalls: api.CapabilityField{Support: api.SupportNo, Reason: "This harness does not emit model-end hook events"},
+			MaxModelCalls: api.CapabilityField{Support: api.SupportYes},
 			MaxDuration:   api.CapabilityField{Support: api.SupportYes},
 		},
 		Telemetry: api.HarnessTelemetryCapabilities{

pkg/hub/handlers.go

@@ -4976,6 +4976,20 @@ func (s *Server) handleProjectRoutes(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
+	// Check for nested /metrics-summary path (lightweight project metrics summary)
+	if subPath == "metrics-summary" {
+		s.handleProjectMetricsSummary(w, r, projectID)
+		return
+	}
+
+	// Check for nested /metrics path (project-scoped metrics dashboard)
+	if subPath == "metrics" || strings.HasPrefix(subPath, "metrics/") {
+		metricsPath := strings.TrimPrefix(subPath, "metrics")
+		metricsPath = strings.TrimPrefix(metricsPath, "/")
+		s.handleProjectMetricsDashboard(w, r, projectID, metricsPath)
+		return
+	}
+
 	// Check for nested /settings path
 	if subPath == "settings" {
 		s.handleProjectSettings(w, r, projectID)