Versioned vector indexing & search for ProllyTree

[zhangfengcdt]

This pull request adds a content-addressed, version-controlled approximate-nearest-neighbour (AKNN) index to ProllyTree, exposed through both the Rust crate and the Python bindings.

The new index lives inside any namespace alongside that namespace's primary key-value tree, so commits, branches, and three-way merges atomically cover both the data and every owned sub-index.

Index shape is a pure function of the current data, so two replicas converge to the same root hash regardless of insertion order. Foundational pieces landed first: a blob storage interface across the supported backends, a back-compat-preserving envelope for large values, and an explicit blob garbage-collection pass.

The text-search layer ships three pluggable embedders — a deterministic dependency-free one for tests, a bundled Candle plus MiniLM embedder for real semantic search, and a callable shim that wraps any Python embedding function.

Documents can be split into chunks at indexing time, with deletes removing every chunk and search results deduplicated back to the document. A cascade mode mirrors primary writes into registered text indexes automatically, and a drift-detection plus repair API recovers from divergence.

[Copilot]

Pull request overview

This PR introduces a version-controlled proximity (vector) index and namespaced text-index/search across the Rust crate and Python bindings, alongside foundational storage work for large-value externalization via content-addressed blobs and explicit blob GC.

Changes:

• Add proximity / proximity_text modules (embedders, chunkers, ANN index, namespaced text index) and expose them through Rust + Python APIs.
• Extend NodeStorage with sync() and a blob API (insert_blob/get_blob/delete_blob/list_blobs) and implement it for in-memory, file, and RocksDB backends.
• Add extensive integration tests, examples, and CI workflows to exercise namespaced stores, text search (including MiniLM), blob externalization, and GC.

Reviewed changes

Copilot reviewed 63 out of 64 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/text_multichunk.rs	Integration tests for multi-chunk text indexing (LineChunker), delete semantics, cascade behavior, and deduped search.
tests/text_index_namespaced.rs	Namespaced text-index tests: reopen checks, embedder mismatch detection, multiple indexes, drift audit/purge.
tests/text_index_minilm.rs	MiniLM embedder integration tests (mostly ignored to avoid HF downloads).
tests/proximity_persistence.rs	Proximity index persistence/roundtrip tests across File (and RocksDB behind feature).
tests/externalization.rs	End-to-end tests for threshold-based externalization and blob GC on file-backed namespaced store.
tests/blob_storage.rs	Backend-agnostic tests for blob insert/get/delete/list semantics and isolation from nodes.
src/storage/mod.rs	Exposes externalize module; extends NodeStorage with sync() and blob API.
src/storage/memory.rs	Implements blob storage for InMemoryNodeStorage (+ unit tests).
src/storage/git.rs	Adds NodeStorage::sync() implementation to flush prolly hash mappings.
src/storage/file.rs	Adds file-backed blob storage (blobs/ dir) + listing support.
src/storage/externalize.rs	Defines envelope wire format + unwrap logic for externalized large values.
src/rocksdb/storage.rs	Adds RocksDB blob keyspace and implements blob CRUD + list.
src/proximity/storage.rs	Bridges ProximityNode storage via a ProllyNode wrapper format.
src/proximity/node.rs	Defines ProximityNode (serialized content-addressed vector buckets).
src/proximity/mod.rs	New proximity module wiring + re-exports (text index, embedders, merge, etc.).
src/proximity/minilm.rs	Candle-backed MiniLM embedder with lazy download/cache + embedding implementation.
src/proximity/level.rs	History-independent level assignment via leading-zero-bits hashing.
src/proximity/embedder.rs	Defines Embedder + deterministic HashEmbedder.
src/proximity/distance.rs	Defines distance metrics (L2, Cosine, InnerProduct) and Distance trait.
src/proximity/chunker.rs	Defines Chunker, IdentityChunker, LineChunker (+ tests).
src/lib.rs	Exposes proximity module behind feature flag.
src/git/versioned_store/mod.rs	Re-exports new namespaced/blob-gc/proximity-related types.
src/git/types.rs	Adds proximity-merge conflict error variant behind feature flag.
python/tests/test_proximity.py	Python integration tests for namespaced text index, cascade, audit, chunking, and feature flags.
python/README.md	Updates doc links and documents namespaces + text search features.
python/prollytree/init.py	Adds feature availability flags and conditional imports for new bindings.
python/examples/text_index_example.py	Runnable Python example for text indexing, cascade, chunking, drift audit/repair, and MiniLM.
python/examples/run_examples.sh	Updates runnable example list to include namespaced + text index examples.
python/examples/requirements.txt	Simplifies example requirements (no longer LangGraph/LangMem dependencies).
python/examples/namespaced_example.py	Runnable Python example demonstrating NamespacedKvStore behavior.
python/docs/README.md	Notes primary docs move to MkDocs; Sphinx kept as fallback/API ref.
python/docs/examples.rst	Removes LangMem example content from Sphinx docs.
python/build_python.sh	Updates --all-features set and fixes wheel selection to pick newest artifact.
pyproject.toml	Enables proximity features in default wheel build feature set.
mkdocs.yml	Adds Text Search to MkDocs navigation.
examples/worktree.rs	Refactors worktree example to a supported pattern and updates merge demonstration.
examples/text_index.rs	New Rust example demonstrating namespaced text index usage, cascade, and chunking.
examples/storage.rs	Ensures tree config persistence is demonstrated correctly when reopening RocksDB storage.
examples/sql.rs	Adjusts commit timing to avoid GlueSQL metadata staleness between read steps.
examples/namespaced.rs	New Rust example demonstrating NamespacedKvStore usage and semantics.
docs/text_search.md	New/expanded documentation page for text indexing & vector search.
docs/quickstart.md	Updates quickstart pointers to include namespaces and text indexing.
docs/index.md	Highlights vector/text search as a top-level capability.
docs/faq.md	Updates AI memory backend FAQ entry (reflect new namespaces/text-search approach).
docs/examples/text_search.md	New worked examples page for text search (Python snippets).
docs/examples/python.md	Removes LangMem section; keeps Python examples focused on core features.
docs/examples/index.md	Adds Text Search to examples index and updates Python bindings blurb.
docs/architecture.md	Adds proximity/text-search architecture section and Python surface updates.
docs/api/python.md	Adds full NamespacedKvStore + text-index API documentation and embedder docs.
Cargo.toml	Adds proximity feature flags, Candle/tokenizers/ureq deps, and new examples.
.github/workflows/rust_examples.yml	New CI workflow to build+run all Rust examples with appropriate features.
.github/workflows/release.yml	Extends wheel release feature sets to include proximity/proximity_text.
.github/workflows/python.yml	Builds Python wheels with proximity/proximity_text features in CI.
.github/workflows/python_examples.yml	New CI workflow to build wheel and run all Python examples (with MiniLM cache).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.