Rebuild index around shared Files and meta-driven snapshots

AGENTS.md

@@ -6,7 +6,11 @@ Guidelines for AI agents working on the sift codebase.
 
 Sift is an indexed code search engine written in Rust, built around **composable on-disk indexes**. It builds indexes tuned to the search workload, then uses them to narrow candidate files before running the full regex engine.
 
-The core architecture treats code search like database query execution: every kind implements one `Index` trait; `Indexes` orchestrates build/update/search and intersects `query` results. File resolution goes through `Plan::resolve`. Today the default index is runtime-width N-gram (trigram default).
+The core architecture treats code search like database query execution:
+`StoreMeta` configures a catalog of `IndexRecord`s; `Indexes` builds snapshots
+and intersects private kind queries. File resolution goes through
+`Plan::resolve`. Today the default index is runtime-width N-gram (trigram
+default).
 
 The candidate pipeline is **plan (pure) → resolve (I/O) → search**: `Plan::new` decides discovery without querying indexes; `Plan::resolve` is the single I/O boundary (query + walk + order); `Searcher` consumes lazy `Candidates` (`into_vec()` materializes all).
 
@@ -40,8 +44,8 @@ evidence for performance PRs.
 |------|------|
 | `crates/core/` | `sift-core`: composable index registry, query planning, candidate narrowing, search engine |
 | `crates/core/src/candidates/` | Index-agnostic candidate description, planning, and resolution |
-| `crates/core/src/index/` | `Index` trait, `IndexRecord`, `Indexes` orchestrator, `Snapshot` |
-| `crates/core/src/index/ngram/` | N-gram `Index` impl (first shipped kind) |
+| `crates/core/src/index/` | `StoreMeta`, `IndexRecord`, `Files`, disk snapshots, `Indexes` |
+| `crates/core/src/index/ngram/` | N-gram kind implementation (first shipped kind) |
 | `crates/core/src/search/` | Query, Searcher, Origin, SearchMode, report/events |
 | `crates/core/src/corpus/` | `File`, `FileFilter`, `FileOrder`, walk |
 | `crates/cli/` | `sift-grep`: `sift` / `sift-daemon` binaries (clap CLI over core) |
@@ -55,11 +59,11 @@ evidence for performance PRs.
 
 | Type | Module | Role |
 |------|--------|------|
-| `Indexes` | `index` | `.sift` directory: meta + current snapshot; `open` / `load` / `build` / `update` |
-| `Snapshot` | `index/snapshot` | Committed opened indexes, lease, and `Files` |
+| `Indexes` | `index` | `.sift` directory: meta + current snapshot; `open` / `load` / `build` |
 | `Files` | `index` | Snapshot-owned `FileId → File` map |
-| `Index` | `index` trait + `ngram` | Opened index only |
-| `IndexRecord` | `index` | Catalog knobs; `build` / `open` → `Box<dyn Index>` |
+| `StoreMeta` | `index` | Persistent corpus, walk, filter, coverage, and catalog configuration |
+| `IndexRecord` | `index` | Typed catalog record; builds kind artifacts and privately opens a kind |
+| `SnapshotId` | `index` | Opaque committed snapshot identity |
 | `Plan` | `candidates` | Pure discovery decision |
 | `Candidates` | `candidates` | Output of `Plan::resolve` |
 | `Query` | `search` | Patterns + options |
@@ -71,7 +75,9 @@ evidence for performance PRs.
 | `IndexJob` | `cli/index` | Resolved index lifecycle; `run` |
 | `Daemon` | `cli/index/daemon` | Background work; modules `ipc`, `watcher`, `refresh` |
 
-Values (not aggregates): `StoreMeta`, `IndexConfig`, `SearchMode`, `StatsMode`, `Scan` / `ScanScope`, `FileFilter`, `FileOrder`, `Coverage`. Printing stays under `cli/format`.
+Values (not aggregates): `StoreMeta`, `SearchMode`, `StatsMode`, `Scan` /
+`ScanScope`, `FileFilter`, `FileOrder`, `Coverage`. Printing stays under
+`cli/format`.
 
 ## Key Conventions
 
@@ -99,21 +105,30 @@ Use short, descriptive kebab-case with a type prefix:
 
 ## Core API Entry Points
 
-`Indexes::open(dir, meta)` (lifecycle) / `Indexes::load(dir) -> Result<Option<Indexes>>` (search) → `build` / `update` → `Plan::resolve` → `Searcher::execute`. CLI: `IndexJob::run` / `SnapshotRefresh::run` for lifecycle; `Run::execute` for search; `Daemon` / `DaemonOrchestrator` for background refresh. See `crates/core/README.md`.
+`Indexes::open(dir, meta)` (lifecycle) / `Indexes::load(dir) ->
+Result<Option<Indexes>>` (search) → `build()` → `Plan::resolve` →
+`Searcher::execute`. CLI: `IndexJob::run` / `SnapshotRefresh::run` for
+lifecycle; `Run::execute` for search; `Daemon` / `DaemonOrchestrator` for
+background refresh. See `crates/core/README.md`.
 
 ## Index layer
 
 | Type | Role |
 |------|------|
-| `Index` trait | Opened kind only: `query` / `coverage` / `all_file_ids` / `update` |
-| `IndexRecord` | Typed catalog knobs; `build` / `open` |
-| `IndexConfig` | Corpus/walk/visibility for a write |
-| `IndexDestination` | Directory or snapshot write target |
-| `Indexes` | Build/update + query/hydrate orchestrator |
+| `StoreMeta` | Persistent corpus, walk, filtering, coverage, and catalog configuration |
+| `IndexRecord` | Typed catalog record; builds kind artifacts and privately opens a kind |
+| `Indexes` | Open/load/build + query/hydrate orchestrator |
 | `Files` | Snapshot-owned `FileId → File` map |
-| `StoreMeta` | Store metadata |
+| `SnapshotId` | Opaque committed snapshot identity |
 
-**Do not add to core:** `from_single`, `Indexes::candidates(Query)`, `reconcile`, `unindexed_hit_paths`, or other caller-specific helpers. Callers compose `Indexes::open`, `indexed_corpus().retain_unindexed`, and `Plan::resolve`.
+`record.rs` owns the private `Opened` enum that dispatches queries. There is no
+public `Index` trait or public `Snapshot` type. Snapshot-root `files.bin`
+(`SIFTFIL2`) is shared by all kinds; kind artifacts live beneath
+`snapshots/<id>/<kind-name>/`.
+
+**Do not add to core:** `from_single`, `Indexes::candidates(Query)`, `reconcile`,
+`unindexed_hit_paths`, or other caller-specific helpers. Callers compose
+`Indexes::open`, `Files::retain_unindexed`, and `Plan::resolve`.
 
 ## Architecture & Design
 
@@ -271,36 +286,13 @@ Examples of **good** names that describe the domain action:
 - `build_index_metadata`
 - `posting_ids` with a `GramMatch` (or similar) argument
 
-When a lifecycle function needs to write to either a directory or a
-snapshot store, use a destination enum instead of `*_to_dir` / `*_into` variants:
-
-```rust
-// Do this:
-fn build(&self, dest: IndexDestination<'_>, config: &IndexConfig<'_>) -> Result<()>;
-
-// NOT this (parallel variants):
-fn build(config, output_dir) -> Result;     // directory
-fn build_into(config, writer, ns) -> Result; // snapshot
-```
-
-## IndexDestination
-
-Index writes take a destination and corpus config:
-
-- `IndexDestination` — `Directory(&Path)` or `Snapshot { writer, namespace }`.
-- `build(dest, config)` / `update(dest, config)` — full corpus from `config`.
-- Open is path-only: `IndexRecord::open(dir, root, corpus_kind)`.
-
-See `crates/core/src/index/artifacts.rs`, `record.rs`, and `ngram/`.
-
 ## Module Organization
 
 Organize modules by domain responsibility, not by Rust item category. Avoid
 catch-all files such as `types.rs`, `traits.rs`, `helpers.rs`, or `utils.rs`
 unless the domain itself is genuinely that narrow. Prefer file/module names that
 describe the behavior or concept they own. Use nested modules when a domain has
-clear subdomains, such as `snapshot/store/disk.rs` and
-`snapshot/store/memory.rs`.
+clear subdomains, such as `index/ngram/storage/`.
 
 ## CLI Crate
 
@@ -357,8 +349,12 @@ Clap parses `*Decl` flag groups; **`Argv` resolves effective runtime values**
 - Prefer printer/JSON rendering via match on `Origin` variants; do not Path-force stream labels for API uniformity.
 - Prefer enums over bools for real alternatives (`Quiet`, `InvertMatch`, `MatchEmissionMode`, `ZeroCounts`).
 - Minimize helper methods as well as free functions—only when absolutely justified.
+- Prefer first-principles entity design: few entities with clear responsibilities; treat extra code and abstractions as liability unless explicitly justified.
+- Keep index orchestration and on-disk storage/versioning index-kind-agnostic; kind-specific logic stays under the kind module (e.g. `ngram/`) so new indexes are easy to add.
+- When planning architecture work, prefer deep critique and a cleaned plan for easy review before implementation.
 
 ## Learned Workspace Facts
 
 - Core search lives under `crates/core/src/search/` (`Query`, `Searcher`, `Origin`, `SearchInputs`, `SearchError`); `Plan` lives under `candidates/`. There is no `sift_core::grep` module or `Grep` facade. The CLI keeps a local `grep` module for `Run`.
 - Daemon IPC is enum-shaped (`DaemonRequest` / `DaemonResponse`); accept loop forwards `Event::Client` — no `FnMut` handler API.
+- Snapshot composition is meant to share one corpus `FileId` → path table per snapshot; kinds return `FileId`s and write only kind artifacts under their namespace.

crates/cli/AGENTS.md

@@ -39,7 +39,9 @@ Scan + Plan::resolve → Candidates
 Searcher::execute → SearchPrinter::print → Report
 ```
 
-Index lifecycle: `IndexJob::run` → `SnapshotRefresh::run` (build/update snapshot). Daemon debouncing and IPC stay in `index/daemon/`.
+Index lifecycle: `IndexJob::run` → `SnapshotRefresh::run` (full rebuild from
+stored metadata). `SnapshotRefresh::run(&mut Indexes)` always calls
+`Indexes::build()`; daemon debouncing and IPC stay in `index/daemon/`.
 
 ## Structure
 

crates/cli/src/index/daemon/mod.rs

@@ -136,7 +136,7 @@ impl DaemonOrchestrator {
         let mut phase = Phase::idle(idle_timeout);
         let mut ingest = IngestTracker::from_reconcile(
             Indexes::open(&sift_dir, &meta)
-                .and_then(|mut indexes| SnapshotRefresh::new(&sift_dir, &meta).run(&mut indexes))?,
+                .and_then(|mut indexes| SnapshotRefresh::run(&mut indexes))?,
         );
 
         let ipc_tx = tx.clone();
@@ -610,7 +610,10 @@ impl DaemonRuntime<'_> {
                     .ok()
                     .and_then(|meta| Indexes::open(self.store.raw, &meta).ok())
                 {
-                    Some(indexes) => indexes.indexed_corpus().retain_unindexed(paths),
+                    Some(indexes) => match indexes.files() {
+                        Some(files) => files.retain_unindexed(paths),
+                        None => paths,
+                    },
                     None => paths,
                 };
                 self.refresh.apply_index(

crates/cli/src/index/daemon/refresh.rs

@@ -66,7 +66,7 @@ impl PendingIndex {
     ) -> Option<ReconcileOutcome> {
         let paths = self.take()?;
         let result = Indexes::open(sift_dir, meta)
-            .and_then(|mut indexes| SnapshotRefresh::new(sift_dir, meta).run(&mut indexes));
+            .and_then(|mut indexes| SnapshotRefresh::run(&mut indexes));
         match result {
             Ok(outcome) => Some(outcome),
             Err(e) => {
@@ -133,9 +133,8 @@ impl IndexRefresh<'_> {
             };
             let mut outcome = None;
             if matches!(scope, RefreshScope::CorpusAndPending) {
-                let result = Indexes::open(&sift_dir, &meta).and_then(|mut indexes| {
-                    SnapshotRefresh::new(&sift_dir, &meta).run(&mut indexes)
-                });
+                let result = Indexes::open(&sift_dir, &meta)
+                    .and_then(|mut indexes| SnapshotRefresh::run(&mut indexes));
                 match result {
                     Ok(committed) => outcome = Some(committed),
                     Err(e) => {

crates/cli/src/index/mod.rs

@@ -1,12 +1,11 @@
 //! Index lifecycle (`sift index build`, `sift index update`) and background refresh.
 
-use std::path::{Path, PathBuf};
+use std::path::PathBuf;
 use std::process::ExitCode;
 
 use sift_core::VisibilityConfig;
 use sift_core::{
-    CorpusMeta, FilterMeta, IndexCoverage, IndexError, IndexRecord, Indexes, SnapshotId, StoreMeta,
-    WalkMeta,
+    CorpusMeta, FilterMeta, IndexCoverage, IndexRecord, Indexes, SnapshotId, StoreMeta, WalkMeta,
 };
 
 use std::str::FromStr;
@@ -159,7 +158,7 @@ impl IndexJob {
             return ExitCode::from(2);
         }
 
-        if let Err(e) = SnapshotRefresh::new(&self.sift_dir, &meta).run(&mut indexes) {
+        if let Err(e) = SnapshotRefresh::run(&mut indexes) {
             eprintln!("sift: {e}");
             return ExitCode::from(2);
         }
@@ -285,52 +284,20 @@ impl IndexJob {
     }
 }
 
-/// CLI orchestration for snapshot build or update.
-pub struct SnapshotRefresh<'a> {
-    sift_dir: &'a Path,
-    meta: &'a StoreMeta,
-}
+/// CLI orchestration for a full snapshot rebuild.
+pub struct SnapshotRefresh;
 
-impl<'a> SnapshotRefresh<'a> {
-    #[must_use]
-    pub const fn new(sift_dir: &'a Path, meta: &'a StoreMeta) -> Self {
-        Self { sift_dir, meta }
-    }
-
-    /// Rebuild or update index files for the full configured corpus.
+impl SnapshotRefresh {
+    /// Rebuild index files for the full configured corpus.
     ///
     /// # Errors
     ///
-    /// Propagates build/update failures from the underlying index kinds.
-    pub fn run(self, indexes: &mut Indexes) -> sift_core::Result<ReconcileOutcome> {
-        let build = self.meta.write_config();
-        let catalog = self.meta.catalog();
-        let (snapshot_id, changed) = if indexes.current_id().is_none() {
-            (SnapshotId::new(indexes.build(catalog, &build)?), true)
-        } else {
-            match indexes.update(catalog)? {
-                Some(id) => (SnapshotId::new(id), true),
-                None => (Self::current_snapshot_id(indexes, self.sift_dir)?, false),
-            }
-        };
+    /// Propagates build failures from the underlying index kinds.
+    pub fn run(indexes: &mut Indexes) -> sift_core::Result<ReconcileOutcome> {
+        let snapshot_id = SnapshotId::new(indexes.build()?);
         Ok(ReconcileOutcome {
             snapshot_id,
-            changed,
+            changed: true,
         })
     }
-
-    fn current_snapshot_id(indexes: &Indexes, sift_dir: &Path) -> sift_core::Result<SnapshotId> {
-        indexes
-            .current_id()
-            .map(|id| SnapshotId::new(id.to_string()))
-            .ok_or_else(|| {
-                sift_core::Error::Index(IndexError::Io {
-                    path: sift_dir.to_path_buf(),
-                    source: std::io::Error::new(
-                        std::io::ErrorKind::NotFound,
-                        "no current snapshot after reconcile",
-                    ),
-                })
-            })
-    }
 }

crates/cli/tests/integration_daemon.rs

@@ -122,9 +122,11 @@ where
 
 fn path_indexed(sift_dir: &Path, rel: &str) -> bool {
     StoreMeta::read(sift_dir).ok().is_some_and(|meta| {
-        Indexes::open(sift_dir, &meta)
-            .ok()
-            .is_some_and(|indexes| indexes.indexed_corpus().contains(Path::new(rel)))
+        Indexes::open(sift_dir, &meta).ok().is_some_and(|indexes| {
+            indexes
+                .files()
+                .is_some_and(|files| files.contains(Path::new(rel)))
+        })
     })
 }
 

crates/core/AGENTS.md

@@ -11,32 +11,33 @@ Indexes::open (lifecycle) / Indexes::load → Option (search)
 Plan::new (pure) → Plan::resolve (query I/O) → Searcher::execute
 ```
 
-- `Indexes` — build/update/publish and query/hydrate over one store (`load` is `None` when absent)
-- `Snapshot` — shared `Files` + opened `Box<dyn Index>` vec for a committed snapshot
+- `Indexes` — builds, publishes, queries, and hydrates one store (`load` is `None` when absent)
+- `Files` — shared `FileId → File` table for the current committed snapshot
 - `Plan` — pure discovery decision; `resolve` owns index query I/O
 - `Searcher` — match execution over resolved candidates and streams
 - `Query` — patterns + options; owns narrowing policy
 - `File` / `Origin` — path identity (`Origin::{File, Stream { label }}`)
 
-Today the default index is `ngram::Index` opened from an `IndexRecord` (trigram width).
+Today the default catalog record is N-gram width 3. `record.rs` privately opens
+kinds through its `Opened` enum.
 
 ## Public API
 
 Search (re-exported from `lib.rs`):
 
 - `Query`, `Searcher`, `Report`, `Origin`, `SearchMode`
-- `Indexes`, `IndexedCorpus`, `SnapshotId`, `Files`
-- `Index`, `IndexRecord`, `IndexConfig`, `IndexDestination`, `ngram::Index`, `GramWidth`
+- `StoreMeta`, `IndexRecord`, `Indexes`, `SnapshotId`, `Files`, `CorpusKind`
+- `ngram::Index`, `GramWidth`, `GramNorm`
 - `Candidates`, `Plan`, `Scan`, `ScanScope`, `SnapshotFreshness`, `Coverage`
 
 ## Source map
 
 | Module | Responsibility |
 |--------|----------------|
-| `index/indexes.rs` | `Indexes` build/update + query/hydrate |
-| `index/record.rs` | `IndexRecord`, opened `Index` |
+| `index/indexes.rs` | `Indexes` open/load/build + query/hydrate |
+| `index/record.rs` | `IndexRecord`, private `Opened` dispatch |
 | `index/files.rs` | Snapshot-owned `Files` |
-| `index/snapshot/` | `Snapshot`, persistence |
+| `index/disk.rs` | Snapshot persistence |
 | `index/ngram/` | N-gram implementation (artifact names live here) |
 | `index/mmap.rs` | Sole `unsafe` in the crate (`mmap_open`) |
 | `search/` | `Query`, `Searcher`, `Report`, events |
@@ -59,7 +60,7 @@ Planning is pure; `Plan::resolve` is the only candidate I/O boundary.
 ## Invariants
 
 - Conservative narrowing: indexes may over-return, never under-return.
-- Multi-index intersection in `Indexes::query`, not per-caller.
+- Multi-kind intersection happens in `Indexes::query`, not per-caller.
 - No free helper functions — logic lives on the owning type.
 - No callback/`FnOnce` APIs.
 - No `unsafe` outside `index/mmap.rs`.