resolve query tables by SQL name (#119)

Author: philcunliffe

README.md

@@ -299,7 +299,8 @@ Freshness is treated asymmetrically (since v1.7.0):
 
 Use `ctvs query refresh <file.jsonl>` to refresh selected source files, or
 `ctvs query refresh --all [dataset]` when you explicitly want the broader
-walk. Use `--refresh always` to force a refresh before the query runs. Use
+walk. Repeat `--date` to query or refresh several UTC date partitions at once.
+Use `--refresh always` to force a refresh before the query runs. Use
 `--strict-freshness` to restore the pre-1.7 behavior where stale partitions
 are a hard error (useful in CI / scheduled jobs that must never read
 outdated data).
@@ -310,7 +311,12 @@ outdated data).
 > unchanged; the new warning is written only to stderr. `missing`
 > partitions still error.
 
-Logical datasets are `logs`, `traces`, `metrics`, `proxy_messages`, and `gascity_messages`. `ctvs collect <file.jsonl> --name <name>` registers an external JSONL file as a dynamic table; `ctvs collect --glob '<pattern>' --name <name>` backs one table with many source files. Names are normalized for SQL, so `--name random-log` becomes table `random_log`. Collection tables include `_ctvs_source_path`, `_ctvs_line_number`, `_ctvs_raw`, and inferred top-level JSON fields. Deleted glob sources remain queryable from their cache-only partitions until the collection is removed. `ctvs query schema <dataset>` prints the schema, and `ctvs query catalog` shows which datasets have source and cached rows.
+Logical datasets are `logs`, `traces`, `metrics`, `proxy_messages`, and `gascity_messages`. `ctvs collect <file.jsonl> --name <name>` registers an external JSONL file as a dynamic table; `ctvs collect --glob '<pattern>' --name <name>` backs one table with many source files. Names are normalized for SQL, so `--name random-log` becomes table `random_log`; quoted SQL can also reference the original collection name as `"random-log"`. Collection tables include `_ctvs_source_path`, `_ctvs_line_number`, `_ctvs_raw`, and inferred top-level JSON fields. Deleted glob sources remain queryable from their cache-only partitions until the collection is removed. `ctvs query schema <table>` prints the schema, and `ctvs query catalog` shows which datasets have source and cached rows.
+
+```bash
+ctvs query sql "select date, count(*) from proxy_messages group by date" \
+  --date 2026-05-14 --date 2026-05-15 --refresh always
+```
 
 ### Conversation log model
 

skills/collectivus-query/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: collectivus-query
-description: Inspect local Collectivus recordings with the ctvs query CLI. Use when the user asks about recorded logs, traces, metrics, LLM proxy exchanges, query cache freshness, or wants SQL over local Collectivus data.
+description: Inspect local Collectivus recordings with the ctvs query CLI. Use when the user asks about recorded logs, traces, metrics, LLM proxy exchanges, query cache freshness, or wants SQL over local Collectivus data, including collected JSONL tables.
 ---
 
 # Collectivus Query
@@ -18,6 +18,7 @@ Use `ctvs query` to inspect local Collectivus recordings. It reads local JSONL r
    - Pass `--strict-freshness` only when the user explicitly needs the pre-1.7 strict mode (e.g., scheduled checks that must never read stale data); it turns stale partitions back into a hard error.
 4. Prefer structured output for analysis: use `--format json` for follow-up reasoning, `--format markdown` when showing a table to the user, and `--limit` to keep output bounded.
 5. Use high-level query commands before custom SQL. Switch to `ctvs query sql` only when the built-in commands cannot answer the question.
+6. For unfamiliar SQL tables, run `ctvs query schema <table> --format json` before querying. It works for built-in recording tables and tables registered with `ctvs collect`.
 
 ## Common Commands
 
@@ -30,6 +31,7 @@ ctvs query logs --since 1h --format json
 ctvs query traces slow --limit 20 --format json
 ctvs query metrics list --format json
 ctvs query metrics series <metric-name> --format json
+ctvs query schema <table> --format json
 ctvs query proxy get <conversation-id> --format json
 ctvs query proxy stats --format json
 ctvs query errors --since 24h --format json
@@ -39,7 +41,9 @@ ctvs collect <file.jsonl> --name <name>
 ctvs collect --glob '<pattern>' --name <name>
 ```
 
-`ctvs collect` registers an external local JSONL file as a dynamic SQL table and immediately refreshes its query cache. Collection names are normalized for SQL (`random-log` -> `random_log`). Query them with `ctvs query sql "select * from random_log"`. Pass `--glob '<pattern>'` instead of a single path to back one logical table with many files; each matched file becomes its own cache partition and `_ctvs_source_path` tells you which file a row came from.
+`ctvs collect` registers an external local JSONL file as a dynamic SQL table and immediately refreshes its query cache. `ctvs query sql` resolves table names from the SQL AST and injects built-in recording tables plus registered collection tables by name. Collection SQL can use either the normalized table name (`random-log` -> `random_log`) or the original quoted collection name (`"random-log"`), for example `ctvs query sql 'select * from "random-log"'`. Pass `--glob '<pattern>'` instead of a single path to back one logical table with many files; each matched file becomes its own cache partition and `_ctvs_source_path` tells you which file a row came from.
+
+Repeat `--date` to query or refresh multiple UTC date partitions at once, for example `ctvs query sql "select count(*) from proxy_messages" --date 2026-05-14 --date 2026-05-15`.
 
 ## Proxy conversation log model
 
@@ -109,9 +113,9 @@ Use `JSON_VALUE(<col>, '$.path')` to extract scalars from the `attributes` / `st
 - Do not assume the cache auto-refreshes. Query commands default to `--refresh never`, and stale partitions return data with a stderr warning rather than refreshing themselves.
 - Always read stderr. A successful exit code does not mean the cache is current — a `warning: query cache last refreshed at …` line on stderr means stdout reflects cache rows from that refresh, and the user should be told before drawing conclusions.
 - Do not paste `--config` into every command by habit. Use it when discovery shows the service is not using `~/.hyp/collectivus.json`.
-- Do not read arbitrary Parquet files directly for `ctvs query sql`; the CLI only allows logical tables.
-- Keep SQL read-only and use only logical datasets: `logs`, `traces`, `metrics`, `proxy_messages`, `gascity_messages`, and registered collection tables from `ctvs query catalog`.
-- Use UTC dates with `--date YYYY-MM-DD`.
+- Do not read arbitrary Parquet or Iceberg files directly for `ctvs query sql`; the CLI resolves SQL table names and injects only known query tables.
+- Keep SQL read-only and use only query tables from `ctvs query catalog`: built-ins (`logs`, `traces`, `metrics`, `proxy_messages`, `gascity_messages`) and registered collection tables.
+- Use UTC dates with `--date YYYY-MM-DD`; repeat `--date` when the user wants a union across multiple date partitions.
 - Use `--service`, `--gateway-id`, `--from`, `--to`, or `--since` to narrow broad investigations.
 
 ## Reference

skills/collectivus-query/agents/openai.yaml

@@ -1,6 +1,6 @@
 interface:
   display_name: "Collectivus Query"
   short_description: "Query local Collectivus recordings"
-  default_prompt: "Use $collectivus-query to inspect local Collectivus logs, traces, metrics, or LLM proxy messages."
+  default_prompt: "Use $collectivus-query to inspect local Collectivus logs, traces, metrics, LLM proxy messages, gascity messages, or collected JSONL tables."
 policy:
   allow_implicit_invocation: true

skills/collectivus-query/references/query-cli.md

@@ -12,7 +12,7 @@ External JSONL collections registered with `ctvs collect <file.jsonl> --name <na
 <recording-root>/.collectivus-query/collections.json
 ```
 
-Their query cache lives under `<recording-root>/.collectivus-query/cache/collections/<table>/source=<hash>/cursor.json`, with rows stored in local Iceberg tables below each source partition. Names are normalized for SQL, so `--name random-log` exposes table `random_log`.
+Their query cache lives under `<recording-root>/.collectivus-query/cache/collections/<table>/source=<hash>/cursor.json`, with rows stored in local Iceberg tables below each source partition. SQL can reference either the normalized table name (`--name random-log` exposes `random_log`) or the original quoted collection name (`"random-log"`).
 
 The cache is explicit. Query commands do not refresh it unless `--refresh always` is passed.
 
@@ -32,7 +32,7 @@ Commands default to `~/.hyp/collectivus.json`. If the running gateway or OTEL co
 - `--cache-dir <dir>`: Override the query cache directory.
 - `--from <timestamp>` / `--to <timestamp>`: Inclusive timestamp bounds.
 - `--since <duration>`: Relative lower bound such as `15m`, `2h`, or `7d`.
-- `--date <YYYY-MM-DD>`: Restrict to one UTC date partition.
+- `--date <YYYY-MM-DD>`: Restrict to one UTC date partition. Repeat it to query or refresh multiple days.
 - `--gateway-id <id>`: Restrict to one gateway id.
 - `--service <name>`: Restrict `serviceName` for logs, traces, and metrics.
 - `--limit <n>`: Maximum rows to render. Default `100`, maximum `1000`.
@@ -47,7 +47,7 @@ Commands default to `~/.hyp/collectivus.json`. If the running gateway or OTEL co
 - `ctvs query doctor`: Check config, recording root, source files, and cache freshness.
 - `ctvs query status`: Inspect source partitions and cache freshness.
 - `ctvs query catalog`: List logical datasets, columns, source partitions, and cached row counts.
-- `ctvs query schema <dataset>`: Print static schema for a logical dataset.
+- `ctvs query schema <table>`: Print schema for a built-in or collected query table.
 - `ctvs query refresh <file.jsonl>... [--force]`: Materialize selected JSONL source files into the query cache.
 - `ctvs query refresh --all [dataset] [--force]`: Materialize all matching JSONL source files into the query cache.
 - `ctvs query sample <dataset>`: Show sample rows.
@@ -69,9 +69,10 @@ Use `ctvs collect` to register arbitrary local JSONL files as dynamic query tabl
 ctvs collect random-log.jsonl --name random-log
 ctvs collect --glob '/path/to/segments/**/*.jsonl' --name segments
 ctvs query sql "select * from random_log" --format json
+ctvs query sql 'select * from "random-log"' --format json
 ```
 
-`ctvs collect` stores the absolute source path (or glob) and immediately refreshes the query cache. If the source file changes later, normal query freshness rules apply: stale cached data is queryable with a stderr warning, `--strict-freshness` turns that into an error, and `ctvs query refresh <file.jsonl>` refreshes selected files. Use `--refresh always` to refresh before running the query.
+`ctvs collect` stores the absolute source path (or glob) and immediately refreshes the query cache. If the source file changes later, normal query freshness rules apply: stale cached data is queryable with a stderr warning, `--strict-freshness` turns that into an error, and `ctvs query refresh <file.jsonl>` refreshes selected files. Use `--refresh always` to refresh before running the query. SQL can reference the normalized table name or the original quoted collection name, such as `"random-log"`.
 
 With `--glob`, one logical table is backed by many source files: each matched file becomes its own cache partition under `.collectivus-query/cache/collections/<table>/source=<hash>/cursor.json`, and refresh appends from each file's recorded cursor when possible. Files that no longer match the glob remain queryable as cache-only partitions. Inside SQL, use `_ctvs_source_path` to see which file a row came from.
 
@@ -85,7 +86,7 @@ Collection tables always include `_ctvs_source_path`, `_ctvs_line_number`, and `
 - `proxy_messages`: One row per LLM proxy content part (text block, tool call, tool result, etc.), globally deduped by content-derived `message_id`. See **proxy_messages columns** below for the full 26-column schema; `gateway_id` and `date` are added as partition columns in the query cache.
 - `gascity_messages`: One row per content block from gascity-captured agent sessions (text, thinking, tool_use, tool_result, attachment). Captured by the `ctvs gascity` supervisor source — agent-attributed (`gascity_template` / `gascity_rig` / `gascity_alias`) and includes per-frame token usage with cache breakdown. Always fresh: the daemon writes Parquet directly to `~/.collectivus/sink/gascity_messages/date=<YYYY-MM-DD>/city=<name>/` (no JSONL stage, no `.meta.json` sidecar). The constant `gateway_id = 'gascity-scribe'` tags the source for cross-source UNIONs with `proxy_messages`. Run `ctvs query schema gascity_messages --format markdown` for the full 47-column schema.
 
-Run `ctvs query schema <dataset> --format json` for the exact columns in the installed version.
+Run `ctvs query schema <table> --format json` for the exact columns in the installed version. Schema lookup works for built-in tables and tables registered with `ctvs collect`.
 
 ## proxy_messages columns
 
@@ -129,6 +130,7 @@ ctvs query sql "select serviceName, count(*) as logs from logs group by serviceN
 ctvs query sql "select traceId, name, durationMs from traces order by durationMs desc limit 20" --refresh always --format json
 ctvs query sql "select model, count(distinct message_id) as messages, count(*) as parts from proxy_messages where role = 'assistant' group by model order by messages desc" --format markdown
 ctvs query sql "select conversation_id, count(distinct message_id) as messages from proxy_messages group by conversation_id order by messages desc limit 10" --format markdown
+ctvs query sql "select date, count(*) as parts from proxy_messages group by date order by date" --date 2026-05-14 --date 2026-05-15 --format markdown
 ```
 
 For JSON columns (`attributes`, `status`, `tools`, `tool_args`), extract scalars with `JSON_VALUE(<col>, '$.path')`:
@@ -137,4 +139,4 @@ For JSON columns (`attributes`, `status`, `tools`, `tool_args`), extract scalars
 ctvs query sql "select model, sum(cast(JSON_VALUE(attributes, '\$.usage.input_tokens') as bigint)) as input_tokens from (select distinct message_id, model, attributes from proxy_messages where role = 'assistant') group by model order by input_tokens desc" --format markdown
 ```
 
-SQL must be a read-only `select` over the logical datasets above or registered collection tables.
+SQL must be a read-only `select` over known query tables. Table names are resolved from the SQL AST and may be built-ins (`logs`, `traces`, `metrics`, `proxy_messages`, `gascity_messages`) or registered collection tables from `ctvs query catalog`.