Merge pull request #2331 from MemPalace/chore/changelog-2322-into-3.8.0

docs(changelog): file the sync corroboration fix under 3.8.0

Author: igorls

CHANGELOG.md

@@ -6,14 +6,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ---
 
-## [Unreleased]
-
-### Bug Fixes
-
-- **`sync --apply` no longer deletes drawers whose source file it could not reach.** `_classify_drawer` separated keep from remove with one `Path.exists()`, and `missing` feeds `removable_ids`, so every state that answered no became a deletion. Of twelve source-file states driven through `sync_palace`, eight were deleted and four of those had really gone: two were a file on a volume that is not mounted, at an empty mount point or at one holding a committed `.gitkeep`, and two were a path that could not be walked at all. Three further states ended the whole run instead, dry run included, and two of those moved with the interpreter as `pathlib` stopped raising. End to end, a project mined from a mounted volume lost every drawer to one `sync --apply` while the volume was away, and the search came back empty once it returned. Removal now asks for corroboration rather than a probe: a file not at its path is removable only while the palace still sees a source of its own, a regular file, in that same directory. A deletion leaves the file's neighbours where they were; an unmounted volume takes all of them at once, and no errno separates the two. Both halves of the verdict are read again at the moment it is formed, because a volume can leave inside one pass and come back inside one. Everything uncorroborated lands in a new bucket, `unresolved`, never added to `removable_ids` and reported the way removals are: counted beside the other buckets, and its source files named in `unresolved_by_source` and printed, with the remainder stated when that list is cut at five. The price: a directory with no surviving known source corroborates nothing, so a file deleted alone from a one-file directory, or a directory's files deleted together, is kept and reported rather than pruned. (#2320)
-
----
-
 ## [3.8.0] — 2026-08-20
 
 Large palaces get fast and stay small: both storage backends lost their palace-wide read paths, a proxied agent session no longer loads a storage stack it never uses, and agents gained a background watcher so coordination stops stalling on nobody listening.
@@ -25,6 +17,7 @@ Large palaces get fast and stay small: both storage backends lost their palace-w
 
 ### Bug Fixes
 
+- **`sync --apply` no longer deletes drawers whose source file it could not reach.** `_classify_drawer` separated keep from remove with one `Path.exists()`, and `missing` feeds `removable_ids`, so every state that answered no became a deletion. Of twelve source-file states driven through `sync_palace`, eight were deleted and four of those had really gone: two were a file on a volume that is not mounted, at an empty mount point or at one holding a committed `.gitkeep`, and two were a path that could not be walked at all. Three further states ended the whole run instead, dry run included, and two of those moved with the interpreter as `pathlib` stopped raising. End to end, a project mined from a mounted volume lost every drawer to one `sync --apply` while the volume was away, and the search came back empty once it returned. Removal now asks for corroboration rather than a probe: a file not at its path is removable only while the palace still sees a source of its own, a regular file, in that same directory. A deletion leaves the file's neighbours where they were; an unmounted volume takes all of them at once, and no errno separates the two. Both halves of the verdict are read again at the moment it is formed, because a volume can leave inside one pass and come back inside one. Everything uncorroborated lands in a new bucket, `unresolved`, never added to `removable_ids` and reported the way removals are: counted beside the other buckets, and its source files named in `unresolved_by_source` and printed, with the remainder stated when that list is cut at five. The price: a directory with no surviving known source corroborates nothing, so a file deleted alone from a one-file directory, or a directory's files deleted together, is kept and reported rather than pruned. (#2320)
 - **`list_drawers` and the tunnel tools stop cold-loading the vector index on a chroma palace.** Both paged `col.get()`, which opens the collection and loads HNSW before answering a question that is pure metadata: on a 1.7 GB / 165k-drawer palace `list_drawers` with no filter and `limit=20` took **36.6 s**, and `find_tunnels` died at 29.7 s with an internal tool error. `chroma.sqlite3` already holds everything those calls need, so they now read it directly and never open the collection. Two traps had to be avoided to make that a win rather than a trade. `chroma:document` lives in `embedding_metadata` alongside the loci, so an unqualified join drags the palace's entire verbatim text — 162 MB across 331k rows here — into memory to render one page; documents are excluded from the scan and the displayed page is hydrated separately. And `embedding_id` is only indexed as the second column of `UNIQUE (segment_id, embedding_id)`, so a join filtering on it alone degenerates into a full scan of `embedding_metadata`: 5.7 s for a single twenty-row page, against 0.003 s once the segment is sought first and `embedding_metadata`'s `(id, key)` primary key second. The wing/room filter is pushed into SQL rather than applied afterwards. Measured on the same palace, `list_drawers(wing=…, limit=20)`: **2.31 s and 1148 MB peak RSS → 0.01 s and 86 MB**. `find_tunnels`' `recent` field survives the move via a `MAX(date)` on the grouped read, and a missing or unreadable palace still reports a diagnostic rather than looking like a palace with no tunnels. (#2314)
 - **`sqlite_exact` search and status stop scanning the whole palace.** `query()` was exact cosine over every row — selecting `id`, `document`, `metadata_json` and `embedding`, JSON-parsing metadata, then dotting in a Python loop — which `mempalace_search` does twice, once for drawers and once for closets. `status` paged every metadata row because the backend had no `facet_counts` and the taxonomy read was chroma-only. Ranking now runs vectorized over the embedding column with only the top-k documents hydrated, the embedding matrix and the `wing` / `room` / `source_file` columns are cached on the long-lived backend handle and invalidated on write, and `facet_counts` plus a grouped `json_extract` read serve status, `list_wings` and `list_rooms`. Exact cosine ranking is unchanged. Measured on a live 167k-drawer / 166k-closet palace: `mempalace_status` **6997 ms → 1045 ms**, warm `mempalace_search` **6364 ms → 210–1600 ms**, wing-scoped search 3910 ms → 1453 ms. (#2308)
 - **The remaining palace-wide reads on `sqlite_exact` are gone.** `get(ids=…)` scanned every row and then did a dict lookup, `include=["metadatas"]` still selected documents and embeddings, and equality filters with `LIMIT`/`OFFSET` were applied in Python — so `list_drawers` at `limit=20` took 2752 ms because it loaded every document in the palace to show twenty previews. Those now go to SQL, pagination walks metadata only to collapse logical drawers and compute the total, and the page's previews are hydrated by id. `status`, `list_wings`, `list_rooms` and `get_taxonomy` share one grouped result behind a 5 s cache dropped on writes, and `graph_stats` gained the same sqlite reconstruction the chroma path uses. (#2311)