feat(observability): run trace tree, /metrics endpoint, hive trace CLI

[chiruu12]

Summary

First observability surface over the structured run logs (master plan A/PR2, unblocked by the goal/step correlation fix that just merged in #55):

• logging/trace.py -- Span model + TraceBuilder deriving a span tree (run -> agent -> goal -> decision/tool) from a run's JSONL logs. Pure data transform: the JSONL files stay the single source of truth, no new write path. Span ids are deterministic; decisions/tools without a goal_id (goal generation, pre-correlation logs) attach to the agent span instead of being dropped.
• GET /runs/{id}/trace -- the span tree as JSON (404 for unknown runs), built off the event loop via asyncio.to_thread like the other log endpoints.
• GET /metrics -- Prometheus text exposition (hand-rendered, no new dependency): agents by status from the store plus latest-run counters (goals generated/completed/abandoned, tool calls, tokens, cost).
• hive trace <run-id> -- renders the tree in the terminal with Rich (goal outcomes, decision token counts, tool success marks).

Test plan

• tests/logging/test_trace.py: tree shape, goal open/close + outcome annotation, deterministic ids, pre-correlation logs still build.
• tests/server/test_rest_api.py: trace 404, /metrics content type + gauge lines.
• Full suite: 1189 passed; ruff, mypy --strict, mkdocs --strict clean.

Docs: endpoint table in rest-api.md, hive trace in cli-reference.md.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Greptile Summary

This PR adds the first observability layer over existing run JSONL logs: a pure-read TraceBuilder that derives a span tree (run → agent → goal → decision/tool), a /runs/{id}/trace JSON endpoint, a hand-rendered /metrics Prometheus endpoint, and a hive trace <run-id> CLI command.

• src/hive/logging/trace.py — new Span model and TraceBuilder; span IDs are deterministic paths, goal outcomes track in_progress until a terminal event, and path segments are percent-encoded to prevent / collisions.
• src/hive/server/routes/system.py — two new routes: /runs/{id}/trace (span tree as JSON) and /metrics (hand-rendered Prometheus text format with agent-status gauges and latest-run snapshot gauges).
• src/hive/cli/main.py — hive trace command renders the span tree with Rich, showing goal outcomes, decision token counts, and tool success marks.

Confidence Score: 5/5

Safe to merge; all changes are additive read-only paths over existing JSONL logs with no new write path.

The TraceBuilder is a pure data transform with no side effects. The two new API endpoints follow the same asyncio.to_thread pattern as the rest of the server. The Prometheus endpoint hand-renders a trivial text format correctly. The two naming issues found are cosmetic and do not affect the correctness of the span tree structure or the API contract.

src/hive/logging/trace.py — the step_index is None dead guard and the goal_id / d{i} span namespace overlap are worth revisiting before the trace format stabilises.

Important Files Changed

Filename	Overview
src/hive/logging/trace.py	New TraceBuilder with deterministic span IDs, in_progress goal tracking, and segment sanitization; the step_index is None guard is dead code (model defaults to 0, not None), and goal IDs matching the d{i}/t{i} naming convention can collide with uncorrelated span IDs.
src/hive/server/routes/system.py	Two clean new endpoints: /runs/{id}/trace (404 on empty list) and /metrics (hand-rendered Prometheus text with correct gauge types and snapshot HELP text).
src/hive/cli/main.py	New hive trace command uses Rich to render the span tree; falls back to logs/ under cwd, consistent with other CLI commands.
tests/logging/test_trace.py	Good coverage: tree shape, open-goal in_progress, deterministic IDs, slash sanitization, and pre-correlation fallback.
tests/server/test_rest_api.py	Adds 404 test for unknown trace and basic Prometheus content-type + gauge line assertions.

Sequence Diagram

sequenceDiagram
    participant CLI as hive trace CLI
    participant API as GET /runs/{id}/trace
    participant Metrics as GET /metrics
    participant TB as TraceBuilder
    participant LR as LogReader
    participant FS as JSONL Logs

    CLI->>TB: build(run_id)
    TB->>LR: get_run(run_id)
    LR->>FS: read run.json
    FS-->>LR: RunLog
    LR-->>TB: "RunLog | None"
    TB->>LR: get_agent_ids(run_id)
    loop per agent
        TB->>LR: get_agent_goals / decisions / tools
        LR->>FS: read goals.jsonl, decisions.jsonl, tools.jsonl
        FS-->>LR: records
        LR-->>TB: "list[GoalLog|DecisionLog|ToolLog]"
    end
    TB-->>CLI: list[Span]
    CLI->>CLI: render Rich tree

    API->>TB: asyncio.to_thread(build, run_id)
    TB-->>API: list[Span]
    API-->>API: model_dump to list[dict]

    Metrics->>LR: asyncio.to_thread(list_runs)
    LR-->>Metrics: list[RunLog]
    Metrics->>LR: asyncio.to_thread(get_summary, runs[0])
    LR-->>Metrics: dict[str, Any]
    Metrics-->>Metrics: render Prometheus text

Loading

_{Reviews (3): Last reviewed commit: "Update src/hive/logging/trace.py" | Re-trigger Greptile}