Skip to content

Commit 5255cf0

Browse files
author
test
committed
Merge remote-tracking branch 'origin/master' into HEAD
2 parents bf43f82 + b1afa7c commit 5255cf0

4 files changed

Lines changed: 292 additions & 24653 deletions

File tree

‎llp/0264-grep-search-mirrors-the-server.decision.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# LLP 0264: `hyp query grep` mirrors the server's grep search, tier for tier
2+
3+
**Type:** Decision
4+
**Status:** Accepted
5+
**Systems:** Query, Cache, CLI, MCP
6+
**Author:** Brendan / Claude
7+
**Date:** 2026-08-18
8+
**Extended-by:** LLP 0265 (implementation plan; its #sequencing resolves the #open item)
9+
**Related:** LLP 0003, LLP 0013, LLP 0034, LLP 0104, LLP 0105, LLP 0209, LLP 0222; hypaware-server LLP 0127, LLP 0128, LLP 0130, LLP 0136, LLP 0157, LLP 0158 (out of tree, design authority for the mechanism)
10+
11+
> Full-text search over recorded sessions ships as `hyp query grep`, a
12+
> read-class verb whose tool is the server's existing `grep_search`. The
13+
> mechanism is the server's, not a new one: mutable data answers by direct
14+
> scan, immutable data answers through hypgrep sidecar indexes, one shared
15+
> column allowlist bounds both, and `--remote` reaches the server's
16+
> archive-backed service unchanged. An earlier draft under this number
17+
> designed a `LIKE` pushdown into the scan engine; the server's shipped
18+
> design replaces it wholesale.
19+
20+
## Context {#context}
21+
22+
Substring search over `ai_gateway_messages` has no fast local path: `LIKE`
23+
is the one predicate the parquet pushdown cannot convert
24+
(hypaware-server LLP 0127 measured a ~13s SQL floor server-side; LLP 0098
25+
here records the same materialize-everything fallback locally).
26+
27+
The server already solved this (hypaware-server `src/search/`): a
28+
grep-search service with **two tiers**. Cache-resident days, which mutate
29+
every tick, are answered by a direct local scan that decodes only the
30+
searchable columns, and are **never indexed** (server LLP 0130: "index
31+
presence is purely a performance property"). Archived day files, which are
32+
immutable, carry `hypgrep` `.index.parquet` sidecars and are answered
33+
through them (server LLP 0128). One allowlist bounds both tiers (server
34+
LLP 0157: `system_text` alone was 90.8% of decoded index-build text, so
35+
"every string column" is refuted by production measurement). Builds run on
36+
a worker thread (server LLP 0136) with a per-file retry bound (server
37+
LLP 0158). The serving surfaces (`POST /v1/search`, the `grep_search` MCP
38+
verb, the admin CLI) are thin wrappers over the one service.
39+
40+
## Decision {#decision}
41+
42+
**The client implements the same mechanism over its own tiers, and the same
43+
tool contract, rather than a new design.** The tier rule transfers, not the
44+
tier names: mutable is scanned, immutable is indexed.
45+
46+
| server | client |
47+
|---|---|
48+
| cache-resident days: direct scan, never indexed | spool + not-yet-compacted data files: direct scan, never indexed |
49+
| archived day files: hypgrep sidecars | compacted cache data files: hypgrep sidecars |
50+
| `SEARCHABLE_COLUMNS` allowlist (LLP 0157) | the same set, imported from the same module |
51+
| `grep_search` MCP tool | the same tool name, compatible `inputSchema` and hit shape |
52+
53+
The client's compacted files qualify as the immutable tier because a data
54+
file is immutable from the moment `stream_append` finalizes it
55+
(`src/core/cache/iceberg/stream_append.js`); compaction and retention
56+
delete whole files, never edit them. Server LLP 0128's rejection of
57+
"index cache files at compaction" is about the *archive* rewriting rows
58+
into different bytes than the cache holds; the client has no second tier
59+
rewriting anything, so the objection does not transfer.
60+
61+
## Shared modules move to core {#shared}
62+
63+
The server already depends on this package (`"hypaware": "file:../hypaware"`).
64+
The pieces both sides must agree on, byte for byte, are hoisted into
65+
hypaware core and imported by the server in a follow-up server PR:
66+
67+
- the searchable-column allowlist and brute-scan projection
68+
(`searchable-columns.js`, server LLP 0157);
69+
- the matcher (literal/regex compile, `test`/`locate`/`rowTest`) and the
70+
snippet window constants;
71+
- the `GrepSearchHit` / `GrepSearchResult` shapes.
72+
73+
Sharing the allowlist is what makes "zero hits" mean the same thing locally
74+
and remotely; two drifting copies would make the same query lie on one side.
75+
76+
## The verb and the tool name {#verb}
77+
78+
`hyp query grep` registers as a read-class **verb** in `CORE_VERBS`,
79+
`tool: 'grep_search'`, with an `inputSchema` compatible with the server's
80+
(`query`, `regex`, `session_id`, `chain_id`, `from`, `to`, `limit`). The
81+
verb is what provides `--remote` (LLP 0034): `hyp query grep --remote
82+
<target>` calls `grep_search` on the server, whose archive fan-out then
83+
serves it with **no server-side feature work**. The verb's summary carries
84+
the server's coverage clause verbatim: only the allowlisted columns are
85+
searched, and zero hits is not evidence the text is absent elsewhere.
86+
87+
**Registration collision, load-bearing.** The server's daemon registers its
88+
own `grep_search` and *defers if the tool already exists*
89+
(hypaware-server `src/daemon.js`, "theirs wins"). Once the kernel ships
90+
this verb, a server booting the new kernel would silently lose its
91+
archive-backed implementation and serve the kernel's local-cache one. The
92+
kernel verb and a server change (register-and-replace, or suppress the core
93+
verb server-side) must land as a coordinated pair; shipping the kernel half
94+
alone is a regression on every server host.
95+
96+
## Local visibility: the one thing the server never faced {#visibility}
97+
98+
`local-only` rows never reach the server (the export seam withholds them,
99+
LLP 0070), so its scan needs no visibility logic. The client scan reads
100+
parquet outside the SQL seam, and the verb lands on the MCP host, one of
101+
the three surfaces [LLP 0105](./0105-query-seam-local-only-visibility.decision.md#surfaces)
102+
names. The scan therefore wraps its per-partition source in the existing
103+
`withLocalOnlyVisibility` wrapper (`src/core/query/visibility.js`), the
104+
same module the SQL seam uses, with the verb's `callerCwd`; the lattice is
105+
not reimplemented. `--include-local-only` carries the LLP 0105 #override
106+
semantics unchanged. Purged rows are handled below the wrapper already:
107+
the icebird source applies position deletes (LLP 0104), and the indexed
108+
path prunes *files*, so a pruned-in row still passes through the
109+
delete-applying read.
110+
111+
## Index lifecycle {#lifecycle}
112+
113+
Sidecars are built during **maintenance/compaction**, the moment a file
114+
becomes immutable, on the worker-thread pattern the server proved
115+
(server LLP 0136), with its per-file retry bound (server LLP 0158) and
116+
sidecar-existence as the idempotency marker (server LLP 0128): no ledger,
117+
a crashed build resumes by listing and skipping. Orphan sweep and
118+
retention delete recursively, so a sidecar dies with its file; a test pins
119+
that. An unindexed file (fresh, raced, or poisoned) is brute-scanned, so
120+
index state is never a correctness input (server LLP 0130's invariant).
121+
122+
## Dependency {#dependency}
123+
124+
`hypgrep` is a plain root `dependency` (the client both builds and reads
125+
indexes). Index writes ride the existing `hyparquet-writer`
126+
optionalDependency exactly as the cache write path does. hypgrep 0.5.1
127+
pins hyparquet 1.27.1, below the 1.28.2 floor
128+
[LLP 0222](./0222-one-pushdown-converter.decision.md#hyparquet-floor)
129+
requires, so adoption includes a root `overrides` entry pinning hypgrep's
130+
hyparquet to 1.28.2 (and its writer to 0.16.6, beside the icebird
131+
override). Widening the range upstream in hypgrep is the durable fix.
132+
133+
## Superseded draft {#superseded-draft}
134+
135+
An earlier, never-committed draft of this decision
136+
(`substring-pushdown-via-ngram-index`) made substring predicates
137+
prunable inside the SQL scan via an icebird row-range hook, with a
138+
predicate whitelist and a `textIndexColumns` registration field. It was
139+
never implemented. The server's shipped service replaces it: same speed
140+
win, no engine surgery, no `appliedWhere` correctness cliff, and one
141+
mechanism across the fleet instead of two.
142+
143+
## Open {#open}
144+
145+
- Sequencing of the server-side PR pair (shared-module import, collision
146+
fix) relative to the client release.

‎llp/0265-grep-search-implementation.plan.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# LLP 0265: `hyp query grep` implementation plan
2+
3+
**Type:** Plan
4+
**Status:** Active
5+
**Systems:** Query, Cache, CLI, MCP
6+
**Author:** Brendan / Claude
7+
**Date:** 2026-08-18
8+
**Related:** LLP 0264 (the decision this executes), LLP 0105 (the visibility wrapper T4 reuses), LLP 0222 (the hyparquet floor T1 must hold), LLP 0034 (the verb surface T5 lands on), LLP 0209 (the compaction pass T6 hooks); hypaware-server LLP 0178 / PR #364 (the server half, already open and inert)
9+
10+
> Turns [LLP 0264](./0264-grep-search-mirrors-the-server.decision.md) into
11+
> seven tasks. The decision is Accepted; the server-side displacement fix
12+
> (hypaware-server #364) is open and mergeable independently.
13+
14+
## Sequencing principle {#sequencing}
15+
16+
**Every wave leaves a shippable tree, and search works before any index
17+
exists.** The scan tier alone is a complete, correct `hyp query grep`
18+
(slower on deep history, never wrong), and `--remote` reaches the server's
19+
archive-backed search from the moment the verb lands, because the server's
20+
`grep_search` tool exists today. So the verb (T5) ships on the scan tier,
21+
and the sidecar build (T6) lands behind it as pure acceleration, exactly
22+
the property server LLP 0130 designed for ("index presence is purely a
23+
performance property").
24+
25+
**Cross-repo order, resolving LLP 0264 #open.** hypaware-server #364
26+
(displace the kernel twin) is inert against today's kernel and merges
27+
first, any time. This repo then implements T1-T7 and releases.
28+
hypaware-server bumps its `hypaware` dependency **last**, after #364 is in,
29+
and follows with its import-swap PR onto the T3 exports. The only forbidden
30+
order is a server dependency bump before #364.
31+
32+
## What was verified against the tree {#verified}
33+
34+
Checked 2026-08-18, each load-bearing for a task below:
35+
36+
- **The server's `grep_search` contract is live** (hypaware-server
37+
`src/daemon.js`): params `query`, `regex`, `session_id`, `chain_id`,
38+
`from`, `to`, `limit`; hits carry `date`, `sessionId`, `agentId`,
39+
`conversationId`, `partId`, `messageId`, `messageCreatedAt`,
40+
`matches: [{column, snippet}]`. T5's schema must be wire-compatible:
41+
`--remote` sends the verb's params to the remote tool verbatim.
42+
- **Neither kernel registry supports removal.** `VerbRegistry` is
43+
register/get/getByTool/list (`src/core/registry/verbs.js`), and the
44+
command registry has no removal either; `registerVerb` also projects a
45+
CLI command immediately. hypaware-server #364 calls
46+
`verbs.unregister(name)` behind a feature guard, so T2 owes exactly that
47+
shape, and it must retract the projected command too.
48+
- **The exports map has `./core/query` but nothing search-shaped**, so T3
49+
adds a `./core/search` entry (types via the `types/` tree like its
50+
siblings).
51+
- **`withLocalOnlyVisibility` wraps any `AsyncDataSource`**
52+
(`src/core/query/visibility.js`), and `storage.dataSourceForTable`
53+
returns the icebird source with position deletes applied (LLP 0104
54+
annotation in `ai-gateway/src/dataset.js`), so T4 gets purge-correct,
55+
visibility-correct rows without reimplementing either.
56+
- **Compaction finalizes immutable files in `compactGeneration`**
57+
(`src/core/cache/maintenance.js`), and the orphan sweep and retention
58+
delete recursively, so a `data/`-adjacent sidecar dies with its file.
59+
T6 builds there and pins the GC with a test rather than writing any.
60+
- **hypgrep 0.5.1 pins hyparquet 1.27.1**, below the exact 1.28.2 floor
61+
([LLP 0222 #hyparquet-floor](./0222-one-pushdown-converter.decision.md#hyparquet-floor)),
62+
and needs `hyparquet-writer` only for `createIndex`; the root already
63+
carries the writer as an optionalDependency on the cache write path.
64+
- **The server's build hardening is directly portable**: worker-thread
65+
handle (`hypaware-server/src/search/index-worker.js`, 149 lines),
66+
three-attempt poison bound (server LLP 0158), sidecar existence as the
67+
idempotency marker (server LLP 0128).
68+
69+
Not verified, and therefore not assumed: whether the verb codec's argv
70+
flag spelling for underscore param names (`--session_id`) is acceptable
71+
CLI ergonomics or T5 needs a codec alias; and the walk cost of a 90-day
72+
cache under the narrow projection (T4 measures it, T6 is sized by it).
73+
74+
## The task graph {#tasks}
75+
76+
### Wave 1 (deps `[]`), three-wide
77+
78+
- **T1, hypgrep dependency + overrides.** `hypgrep` into `dependencies`;
79+
`overrides` pinning its `hyparquet` to 1.28.2 and `hyparquet-writer` to
80+
0.16.6, beside the icebird override. Verify one deduped hyparquet copy
81+
(`npm ls hyparquet`) and that `npm pack --dry-run` stays sane.
82+
Complexity 1.
83+
- **T2, `unregister(name)` on the verb registry.** Removes the verb from
84+
both maps and retracts the projected CLI command, which needs a matching
85+
removal on the command registry (including alias cleanup). This is the
86+
affordance hypaware-server #364 already guards on; its contract is
87+
by-name, idempotent, and safe on an unknown name. Complexity 2.
88+
- **T3, hoist the shared search modules.** `src/core/search/`: the
89+
searchable-column allowlist and brute-scan projection (server
90+
`searchable-columns.js`, imported semantics unchanged), the matcher
91+
(literal/regex compile, `test`/`locate`/`rowTest`) and snippet-window
92+
constants, and the hit shapes as a `.d.ts`. New `./core/search` export.
93+
The server's import swap is server-repo work after this publishes, not
94+
part of this plan. Complexity 2.
95+
96+
### Wave 2 (deps `[T1, T3]`)
97+
98+
- **T4, the local grep service.** `src/core/search/grep_service.js`: flush
99+
the spool first (the query seam's freshness move), walk cache partitions
100+
newest-day-first, per-partition source from `storage.dataSourceForTable`
101+
wrapped in `withLocalOnlyVisibility` with the caller's `cwd`, files
102+
processed sequentially with the narrow projection for unindexed files
103+
and `parquetFind` through the sidecar for indexed ones, stop at the
104+
limit, report `truncated`/`exhausted` like the server. The indexed path
105+
is dormant until T6 but is exercised in tests with a hand-built sidecar.
106+
Complexity 3.
107+
108+
### Wave 3 (deps `[T2, T4]`)
109+
110+
- **T5, the verb.** `src/core/search/grep_verb.js` registered in
111+
`CORE_VERBS`: `name: 'query grep'`, `tool: 'grep_search'`, read-class,
112+
wire-compatible schema (T5 verifies the argv spelling question above),
113+
the server's coverage clause in the summary, `--include-local-only`
114+
with LLP 0105 #override semantics, rg-style snippet render honoring
115+
LLP 0225 (table escapes, `--format json` does not). After this task
116+
`hyp query grep` works locally (scan tier) and `--remote <target>`
117+
works against any current server. Complexity 2.
118+
119+
### Wave 4 (deps `[T1, T3]`, parallel to T4/T5)
120+
121+
- **T6, sidecar build at maintenance.** Port the server's worker handle
122+
and poison bound; `compactGeneration` queues an index build for each
123+
finalized data file over the allowlist columns; sidecar existence is
124+
the marker, no ledger; a test pins that orphan sweep and retention
125+
delete sidecars with their files, and that an unindexed or quarantined
126+
file is served by the scan tier. Complexity 3.
127+
128+
### Wave 5 (deps `[T5, T6]`)
129+
130+
- **T7, surfaces and proof.** `hyp query status` gains an index-coverage
131+
line; `hypaware-query` SKILL.md adds grep (fixing "these are the only
132+
subcommands"), the sub-`ngramLength` literal cliff, and the coverage
133+
caveat; a hermetic smoke (`query_grep_roundtrip`) asserts a scan-tier
134+
hit, a sidecar-tier hit, a purged row absent, and a `local-only` row
135+
withheld from a `full`-class caller and present with the override.
136+
Complexity 2.
137+
138+
## Out of scope {#out-of-scope}
139+
140+
- The server's import swap and dependency bump (hypaware-server repo,
141+
after T3 publishes and #364 merges).
142+
- Any config knob for indexed columns or index cadence: the allowlist is
143+
the shared constant by decision (LLP 0264 #shared), and cadence is
144+
compaction's.
145+
- Remote-side behavior changes: `--remote` intentionally hits the
146+
server's existing service unchanged.

0 commit comments

Comments
 (0)