Add log-driven development skill bundle

Author: philcunliffe

README.md

@@ -362,15 +362,17 @@ their cross-source joins with `proxy_messages`.
 
 ### LLM skill
 
-Install the bundled `collectivus-query` skill so Claude Code and Codex know how
-to inspect local recordings with `ctvs query`:
+Install the bundled Collectivus skills so Claude Code and Codex know how to
+inspect local recordings with `ctvs query` and build log-driven development
+loops:
 
 ```bash
 ctvs skills install --client all
 ```
 
-The skill assumes the default `~/.hyp/collectivus.json` config unless the agent
-discovers a non-default service config from `ctvs status` or the service unit.
+The query skill assumes the default `~/.hyp/collectivus.json` config unless the
+agent discovers a non-default service config from `ctvs status` or the service
+unit.
 
 ## OTLP receiver
 
@@ -627,7 +629,7 @@ supported in this build.
 | `ctvs export --config <path> [...]` | Convert recorded JSONL to local Parquet without invoking the upload scheduler |
 | `ctvs query <command> [...]` | Query local recordings through the explicit query cache |
 | `ctvs collect <file.jsonl>\|--glob <pattern> --name <name>` | Register external JSONL as a dynamic query table |
-| `ctvs skills install [--client claude\|codex\|all]` | Install the bundled Collectivus query LLM skill |
+| `ctvs skills install [--client claude\|codex\|all]` | Install the bundled Collectivus LLM skills |
 
 If stdin is not a TTY, `install` refuses to guess: pass `--yes` to attach
 Claude Code unattended, or `--no` to skip the attach step.

ctvs-vs-claude-code-logs.md

@@ -0,0 +1,177 @@
+# ctvs proxy logs vs Claude Code logs
+
+**Date:** 2026-05-18
+**Author:** investigation captured by Claude (Opus 4.7) working with phil
+**Context:** Investigating whether the Gas City supervisor can surface reasoning tokens from Claude sessions. The investigation expanded into a comparison of three vantage points: Claude Code's on-disk JSONL, the Gas City supervisor's HTTP API, and the ctvs HTTPS proxy capture.
+
+---
+
+## TL;DR
+
+ctvs captures the full HTTPS traffic between Claude Code and Anthropic's API. Claude Code's JSONL captures the post-processed conversation it chose to persist locally. **ctvs is essentially a superset on the request side**, plus it captures wire metadata Claude Code never sees. Claude Code adds a few client-side artifacts ctvs doesn't have (hook output, file snapshots, DAG structure).
+
+Crucially, ctvs is the only layer with visibility into **what Claude Code told Anthropic about your project** — system prompt, tool schemas, request parameters, beta headers, context-management strategies.
+
+---
+
+## What ctvs has that Claude Code JSONL does not
+
+### 1. The exact system prompt as sent to the API
+
+Every API call carries a system prompt that Claude Code constructs at request time:
+
+- Claude Code identity + safety rules
+- Tool-use guidance and conventions
+- Your project's `CLAUDE.md` / `AGENTS.md` content
+- Auto-memory blob from `~/.claude/projects/.../memory/`
+- Git status snapshot at conversation start
+- Environment block (platform, shell, working directory, model ID)
+
+ctvs's `proxy_messages.system_text` column has the full ~30 KB blob per request, prefixed with a billing header:
+```
+x-anthropic-billing-header: cc_version=2.1.143.e33; cc_entrypoint=cli; cch=fe500;
+
+You are Claude Code, Anthropic's official CLI for Claude.
+...
+```
+
+The Claude Code JSONL has none of this. Your project's user instructions ARE visible inside the system prompt (so the prompt content is reconstructible), but the framing/guardrail prose around them is not in the JSONL anywhere.
+
+### 2. Tool definitions as sent
+
+ctvs has the full tool JSON Schema list per request: every tool's `name`, `description`, parameter schemas, examples — the exact contract Claude Code declared to the API.
+
+Claude Code logs only show tool *invocations* (`tool_use` blocks with `{name, input}`), not the schemas. If you want to know which subagent types were available, which Bash flags were enabled, what each tool's description was — that's only in ctvs.
+
+### 3. Request envelope and parameters
+
+Captured under `proxy_messages.attributes.request`:
+
+- `model` — exact model ID used
+- `max_tokens` (64,000 in current captures)
+- `thinking.type` — `adaptive` / `enabled` / `disabled`
+- `output_config.effort` — `low` / `medium` / `high` / `xhigh` / `max` / `auto`
+- `context_management.edits` — including the `clear_thinking_20251015` strategy responsible for thinking redaction on Opus 4.7
+- `stream: true`
+- Beta headers requested (e.g. `context-management-2025-06-27`)
+- `metadata.user_id` — `{device_id, account_uuid, session_id}` (your Anthropic account identifiers)
+
+None of this exists in the Claude Code JSONL. If you want to know **why** thinking is empty in your logs, the answer is in `attributes.request.context_management` — and only ctvs has it.
+
+### 4. Wire-level metadata
+
+- `latency_ms` per request (4,867 ms on a sample)
+- `provider_raw` — Anthropic API response headers / envelope
+- HTTP status, retries (via `ctvs query proxy events`)
+- OTLP traces and metrics (separate datasets in ctvs)
+
+### 5. Per-message billing detail
+
+Both layers have token counts. ctvs reflects what was *actually billed* server-side, including server-tool calls (`web_search_requests`, `web_fetch_requests`, etc.).
+
+### 6. Cross-session conversation continuity
+
+ctvs has a `conversation_id` that can span multiple Claude Code session resumes. Claude Code's JSONL splits on its own session ID; `claude --resume` produces a new file. ctvs stitches them by API request lineage, so you can reconstruct a full multi-day conversation as a single thread.
+
+### 7. Plaintext reasoning, when the API returns it
+
+Both layers happen to have plaintext when the model produces it (e.g., 163/163 Opus 4.6 thinking blocks had plaintext in both Claude Code JSONL and ctvs proxy capture). But ctvs is independent of Claude Code's choice to persist. If a future Claude Code version starts stripping thinking from JSONL even when the API returns it — or if you swap Claude Code for a custom client that doesn't write JSONL at all — ctvs would still capture it.
+
+---
+
+## What Claude Code JSONL has that ctvs does not
+
+The relationship isn't strictly "ctvs is a superset." Claude Code adds client-side artifacts ctvs has no visibility into:
+
+| Field | What it carries |
+|---|---|
+| `uuid` / `parentUuid` | Conversation DAG — branching, sidechains, retries |
+| `compactMetadata` | When and how context was auto-compacted |
+| `attachment.hookName` | `SessionStart:startup` hook stdout/stderr |
+| `file-history-snapshot` | Claude Code's internal snapshots of file state |
+| `ai-title` | Model-generated session title |
+| `permission-mode` | When permission state flipped during the session |
+| `cwd` / `gitBranch` per entry | Local state at each message timestamp |
+| Uploaded attachments | Files the user dropped into the session UI |
+
+Claude Code also has DAG semantics — it lets you reconstruct which response branched from which user message, including ones that were retried or abandoned. ctvs sees a flat sequence of API calls.
+
+---
+
+## Side-by-side: where each layer lives
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  User                                                           │
+│  ↓                                                              │
+│  Claude Code CLI (2.1.143)                                      │
+│  ↓ writes →  ~/.claude/projects/{slug}/{id}.jsonl               │
+│  ↓                                                              │
+│  HTTPS → ←  CAPTURED BY ctvs PROXY (127.0.0.1:8787)             │
+│  ↓                              ↓                               │
+│  Anthropic API                  /Users/phil/.hyp/collectivus/   │
+│                                 phil/proxy/YYYY-MM-DD.jsonl     │
+│                                                                 │
+│  Separately, Gas City supervisor (127.0.0.1:8372) serves the    │
+│  Claude Code JSONL files via                                    │
+│    GET /v0/city/{name}/session/{id}/transcript[?format=raw]     │
+│  It reads the same files Claude Code wrote — no extra data.     │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+| Layer | What it stores | Reasoning visible? |
+|---|---|---|
+| **Anthropic API** | Server-side state | Only Anthropic sees raw reasoning |
+| **ctvs proxy capture** | Full request/response over the wire | When API returns plaintext (model-dependent) |
+| **Claude Code JSONL** | Conversation turns + client artifacts | When Claude Code chose to persist it (model + version-dependent) |
+| **Gas City supervisor** | Pass-through of Claude Code JSONL | Same as Claude Code JSONL |
+
+---
+
+## Why this distinction matters
+
+If you want to know **what Claude said** → either layer works (when reasoning is present at all).
+
+If you want to know **what Claude Code told Anthropic about your project** → only ctvs has it. That includes:
+- The full system prompt with your `CLAUDE.md` injected
+- Every tool schema Claude Code advertised
+- The `clear_thinking_20251015` context-management strategy that strips thinking
+- Beta API flags Claude Code opted into
+- Your Anthropic account identifiers
+- Wire timings and retries
+
+If you want **client-side state** (file snapshots, hook output, permission changes, DAG branching) → only Claude Code JSONL has it.
+
+If you want **cross-session conversation continuity** → only ctvs's `conversation_id` stitches multi-resume sessions together.
+
+---
+
+## Concrete columns reference
+
+### ctvs `proxy_messages` schema (28 columns)
+
+```
+gateway_id, schema_version, conversation_id, user_id, provider, model,
+system_text, tools, conversation_started_at, conversation_source,
+cwd, git_branch, message_id, previous_message_id, message_index,
+message_created_at, role, part_id, part_index, part_type,
+content_text, tool_name, tool_call_id, tool_args, thinking_signature,
+status, attributes, date
+```
+
+### Claude Code JSONL top-level fields (representative)
+
+```
+uuid, parentUuid, type, subtype, message{role, content[], model, usage},
+toolUseID, logicalParentUuid, compactMetadata, isCompactSummary,
+timestamp, sessionId, cwd, gitBranch, version, requestId, userType,
+entrypoint, attachment{hookName, content, stdout, stderr, ...}
+```
+
+The overlap (model, role, message content, timestamps, tool calls, tool results) is what most users see. The non-overlap is where ctvs's value lives.
+
+---
+
+## Footnote: how this report was produced
+
+Live `testcity` supervisor at `127.0.0.1:8372`. ctvs daemon at `127.0.0.1:8787` with proxy recordings under `~/.hyp/collectivus/phil/proxy/`. Counts and samples queried via `ctvs query sql` against the materialized parquet cache. Cross-checked against raw JSONL at `~/.claude/projects/-Users-phil-workspace-gascity/eaec666c-b3ce-46d7-b2c9-66e9f67d65e5.jsonl` (the file backing this very conversation).

skills/log-driven-development/SKILL.md

@@ -0,0 +1,43 @@
+---
+name: log-driven-development
+description: Instrument applications and embedded LLM-agent workflows with Collectivus OTEL and local JSONL so Claude Code or Codex can self-test, query logs/traces/metrics, and iteratively fix issues. Use when building or modifying apps, adding smoke tests, debugging development loops, adding OpenTelemetry, capturing agent events, or using ctvs query/ctvs collect to close a log-driven feedback loop.
+---
+
+# Log Driven Development
+
+## Overview
+
+Treat telemetry as part of the development harness. Add instrumentation early, run a narrow smoke path with a unique run id, inspect Collectivus recordings, then fix or improve the app based on observed behavior.
+
+## Core Loop
+
+1. Pick a stable `service.name` and a per-run id such as `DEV_RUN_ID=smoke-<utc>-<pid>`.
+2. Confirm the Collectivus OTLP endpoint before wiring code. Default local OTLP/HTTP is `http://127.0.0.1:4318`; signal endpoints are `POST /v1/traces`, `POST /v1/metrics`, and `POST /v1/logs`.
+3. Instrument liberally around entrypoints, state transitions, branches, external calls, background jobs, tool calls, retries, validation checks, and error paths.
+4. Add or update a smoke command that exercises the smallest complete workflow and emits telemetry for each step and assertion.
+5. Run the smoke command, query the resulting `logs`, `traces`, `metrics`, and collected JSONL tables, then make the next code change from evidence rather than guesses.
+6. Repeat until the smoke path succeeds and the telemetry shows the expected internal sequence, not just a passing exit code.
+
+## Instrumentation Bias
+
+Prefer broad, cheap, local-first instrumentation over sparse "production-quality" telemetry while developing. It is acceptable for dev telemetry to be chatty when it is guarded by environment flags, avoids secrets, and can be disabled or sampled before production.
+
+Use structured attributes that are easy to query: `dev_run_id`, `smoke_name`, `smoke_step`, `component`, `operation`, `status`, `error_kind`, `tool_name`, `agent_id`, and domain ids. Prefer snake_case for custom attributes so SQL JSON paths stay simple. Keep high-cardinality or large values in local JSONL or short hashes/excerpts, not every OTEL span.
+
+Do not record secrets, credentials, raw customer data, or private prompts by default. For LLM agents, record decisions, tool calls, model settings, token counts, latency, validation outcomes, content hashes, and short redacted excerpts; do not record hidden chain-of-thought.
+
+## Reference Loading
+
+- Read [references/collectivus-otel.md](references/collectivus-otel.md) when adding OpenTelemetry libraries, manual OTLP emitters, or Collectivus-specific endpoint configuration.
+- Read [references/agent-jsonl.md](references/agent-jsonl.md) when the application contains LLM agents, planners, routers, tool callers, evaluators, or other model-driven flows that should write local JSONL for `ctvs collect`.
+- Read [references/self-test-loop.md](references/self-test-loop.md) when adding the smoke command or querying the captured data after a run.
+
+If the `collectivus-query` skill is available, use it for detailed `ctvs query` cache freshness and SQL rules. Otherwise run `ctvs query status`, prefer `--format json` for analysis, and refresh the exact source file when the CLI reports a missing cache partition.
+
+## Done Criteria
+
+- The app can run with telemetry enabled using documented env vars or dev config.
+- Each smoke run has a queryable `dev_run_id` across logs/spans/metrics and any local agent JSONL table.
+- The smoke test checks both external behavior and captured internal signals.
+- Failures produce enough telemetry to identify which step, component, or agent decision failed.
+- The final response mentions the smoke command and the key `ctvs query` or `ctvs collect` command used for verification.

skills/log-driven-development/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "Log Driven Development"
+  short_description: "Instrument apps and close feedback loops"
+  default_prompt: "Use $log-driven-development to instrument this app with Collectivus OTEL and JSONL feedback loops, then smoke test by querying the captured logs."
+policy:
+  allow_implicit_invocation: true

skills/log-driven-development/references/agent-jsonl.md

@@ -0,0 +1,92 @@
+# Local JSONL For LLM Agents
+
+Applications that embed LLM agents should write a local append-only JSONL event stream in addition to OTEL. OTEL gives causality and latency; JSONL preserves agent-specific event shape that is easier to query with `ctvs collect`.
+
+## File Layout
+
+Use a local, gitignored path:
+
+```text
+.collectivus/agents/<agent-id>/<YYYY-MM-DD>.jsonl
+```
+
+If the repo uses another runtime/log directory, follow that convention, but keep files local, append-only, and easy to glob. Add the path to `.gitignore` unless the repo already ignores the parent directory.
+
+Register the stream:
+
+```bash
+ctvs collect --glob '.collectivus/agents/**/*.jsonl' \
+  --name app-agent-events \
+  --timestamp-column timestamp \
+  --replace
+ctvs query schema app_agent_events --format markdown
+```
+
+`ctvs collect` infers top-level fields as SQL columns. Keep common fields top-level and put rare or nested details in `metadata`.
+
+## Event Shape
+
+Use one JSON object per line. Keep field names stable.
+
+```jsonl
+{"timestamp":"2026-05-18T19:00:00.000Z","dev_run_id":"smoke-20260518T190000Z-123","agent_id":"planner","conversation_id":"conv-abc","turn_id":"turn-001","event_type":"tool_call","phase":"act","status":"started","model":"claude-opus-4-7","provider":"anthropic","tool_name":"search_inventory","tool_call_id":"toolu_123","latency_ms":null,"input_sha256":"a1b2","output_sha256":null,"content_excerpt":"search inventory for sku prefix","metadata":{"route":"inventory.lookup"}}
+```
+
+Recommended top-level fields:
+
+- `timestamp`: ISO-8601 UTC time. Use this with `--timestamp-column timestamp`.
+- `dev_run_id`: matches the smoke/test run id used in OTEL.
+- `agent_id`: stable logical agent name.
+- `conversation_id`, `turn_id`, `parent_turn_id`: correlate multi-turn flows.
+- `event_type`: `run_start`, `prompt_built`, `model_request`, `model_response`, `tool_call`, `tool_result`, `decision`, `validation`, `run_end`, `error`.
+- `phase`: `plan`, `act`, `observe`, `reflect`, `evaluate`, or a domain-specific phase.
+- `status`: `started`, `ok`, `error`, `skipped`, `retrying`.
+- `model`, `provider`: model metadata when applicable.
+- `tool_name`, `tool_call_id`: tool correlation when applicable.
+- `latency_ms`, `prompt_tokens`, `completion_tokens`, `total_tokens`: numeric cost/performance fields.
+- `input_sha256`, `output_sha256`: stable hashes for large or sensitive content.
+- `content_excerpt`: short redacted excerpt for local debugging.
+- `error_kind`, `error_message`: bounded error detail.
+- `metadata`: nested JSON for rare details.
+
+## What To Log
+
+Log:
+
+- Agent run start/end with inputs summarized by id/hash.
+- Prompt/template version, selected tools, model, temperature, max tokens, and safety settings.
+- Retrieval/search query summaries, hit counts, chosen ids, and score ranges.
+- Tool call arguments after redaction, tool results summarized by ids/counts/status, and latency.
+- Model response metadata, finish reason, token usage, and structured output validation results.
+- Decisions and routing choices as explicit data: `decision`, `alternatives`, `reason_code`, `confidence`.
+- Evaluator or smoke-test assertions: expected, actual summary, pass/fail, and linked domain ids.
+- Errors, retries, cancellations, and fallbacks.
+
+Do not log hidden chain-of-thought. When reasoning is useful, log a brief decision rationale or machine-readable reason code instead.
+
+## Writer Requirements
+
+The writer should:
+
+- Create parent directories.
+- Append exactly one compact JSON object plus `\n` per event.
+- Avoid partial lines where possible.
+- Never throw in the main application path because logging failed.
+- Redact secrets before writing.
+- Include `dev_run_id` on every event when available.
+
+In tests and local smokes, assert that at least one event was written for each expected agent phase. Then use `ctvs collect` and SQL to inspect the same file from the outside.
+
+## Query Examples
+
+```bash
+ctvs query sql "select event_type, status, count(*) as events from app_agent_events group by event_type, status order by events desc" --format markdown
+```
+
+```bash
+ctvs query sql "select timestamp, agent_id, phase, event_type, status, tool_name, error_message from app_agent_events where dev_run_id = '<run-id>' order by timestamp" --refresh always --format json
+```
+
+```bash
+ctvs query sql "select agent_id, avg(latency_ms) as avg_latency_ms, count(*) as calls from app_agent_events where event_type in ('model_response', 'tool_result') and latency_ms is not null group by agent_id order by avg_latency_ms desc" --format markdown
+```