feat: MCP server v2, smart embedding defaults, incremental build fixes (#279)

feat: MCP server v2 — build, status tools, smart embedding defaults, and incremental build fixes

Author: yichuan-w

.gitignore

@@ -121,3 +121,6 @@ test-code/
 localtestmcp/
 *.csv
 *.pickle
+
+# Personal dev notes (not tracked)
+docs/dev/

.vscode/settings.json

@@ -18,5 +18,10 @@
     "**/*.egg-info/**": true,
     "**/build/**": true,
     "**/dist/**": true
-  }
+  },
+  "accessibility.signals.terminalBell": {
+    "sound": "on",
+    "announcement": "auto"
+  },
+  "cmake.sourceDirectory": "/Users/yichuan/Desktop/code/LEANN/leann/packages/leann-backend-hnsw"
 }

docs/CHANGELOG.md

@@ -3,6 +3,28 @@
 Append-only log of major changes to LEANN (new features, breaking changes, important
 fixes). Newest entries at the bottom.
 
+## 2026-03-05: IVF backend incremental update support
+
+- Added `leann-backend-ivf` with FAISS IndexIVFFlat + DirectMap.Hashtable.
+- IVF supports in-place `add_vectors` and `remove_ids` without full rebuild.
+- `leann build` is now idempotent: re-running on an existing index does incremental update (add new, remove deleted, re-index modified files).
+- Fixed incremental build chunking inconsistency and shared metadata dict bug.
+- Fixed IVF incremental update duplicate chunks from stale `passages.jsonl`.
+
+## 2026-03-05: MCP server v2 — build, status, and structured search
+
+- Added `leann_build` MCP tool: build or incrementally update indexes directly from Claude Code.
+- Added `leann_status` MCP tool: inspect index details (backend, embedding model, chunk/file count, size).
+- `leann_search` now uses `--json` output with file paths always included, formatted as markdown code blocks.
+- Fixed `float32` JSON serialization bug in `leann search --json`.
+- Cleaned up MCP tool descriptions (concise, no emoji).
+
+## 2026-03-05: Documentation — roadmap, vision, and dev guidelines
+
+- Rewrote `docs/roadmap.md` with current P0/P1 priorities from GitHub issue #237.
+- Added `docs/ultimate_goal.md` — long-term vision (personal data platform, best code retrieval MCP, multimodal, local-first).
+- Added self-contained documentation principle and dev doc maintenance rules to `CLAUDE.md`.
+
 ## 2026-06-02: GPU FlashLib IVF backend (`flashlib_ivf`)
 
 - Add `leann-backend-flashlib-ivf`, a GPU IVF-Flat (inverted file) approximate-NN

docs/issue-proposals/smart-embedding-default.md

@@ -0,0 +1,41 @@
+# Smart default embedding model based on platform and corpus size
+
+## Summary
+
+Propose platform- and corpus-aware default embedding model selection for `leann build` when `--embedding-model` is not explicitly specified. This would improve out-of-the-box experience for different deployment scenarios (macOS CPU, NVIDIA GPU, etc.) without changing behavior when users pass an explicit model.
+
+## Motivation
+
+- **Current default**: `facebook/contriever` (~420MB, 768 dim) — heavy for CPU-only builds on large corpora
+- **macOS users** often hit slow builds on 20K+ chunks; lighter models like `all-MiniLM-L6-v2` (~90MB) are much faster
+- **NVIDIA GPU users** can leverage stronger models; smaller corpora benefit from quality (e.g. Qwen3-Embedding-0.6B), larger ones from balanced models (e.g. bge-base-en-v1.5)
+
+## Proposed logic
+
+| Platform | Chunk count | Default model |
+|----------|-------------|---------------|
+| **macOS** | ≥ 20,000 | `sentence-transformers/all-MiniLM-L6-v2` |
+| **macOS** | < 20,000 | `intfloat/e5-small-v2` |
+| **NVIDIA GPU** | < 5,000 | `Qwen/Qwen3-Embedding-0.6B` |
+| **NVIDIA GPU** | ≥ 5,000 | `BAAI/bge-base-en-v1.5` |
+| **Other** | any | `facebook/contriever` (unchanged) |
+
+## Implementation notes
+
+1. **Platform detection**: `torch.cuda.is_available()` for NVIDIA; `sys.platform == "darwin"` for macOS
+2. **Chunk count**: Known only after loading/chunking; may need to either:
+   - Do a lightweight pre-scan (e.g. file count × rough chunks per file), or
+   - Defer default choice until after first chunking pass (and cache for incremental)
+3. **Explicit override**: If user passes `--embedding-model`, always use it; this logic applies only when the flag is omitted
+
+## Model references
+
+- `sentence-transformers/all-MiniLM-L6-v2`: ~90MB, 384 dim, fast on CPU
+- `intfloat/e5-small-v2`: ~90MB, 384 dim
+- `Qwen/Qwen3-Embedding-0.6B`: 0.6B params, 1024 dim, strong retrieval
+- `BAAI/bge-base-en-v1.5`: ~110M params, 768 dim, good MTEB scores
+
+## Open questions
+
+- Should we add a `--embedding-model auto` to explicitly opt into this logic?
+- Pre-scan vs post-chunk decision: trade-off between accuracy and implementation complexity

packages/leann-core/src/leann/api.py

@@ -379,7 +379,7 @@ class LeannBuilder:
     def __init__(
         self,
         backend_name: str,
-        embedding_model: str = "facebook/contriever",
+        embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2",
         dimensions: Optional[int] = None,
         embedding_mode: str = "sentence-transformers",
         embedding_options: Optional[dict[str, Any]] = None,

packages/leann-core/src/leann/cli.py

@@ -31,6 +31,28 @@
 from .sync import DEFAULT_INDEX_EXTENSIONS, FileSynchronizer, parse_include_extensions
 
 
+def _default_embedding_model() -> str:
+    """Pick a sensible default embedding model based on platform.
+
+    | Platform   | Default model                                  |
+    |------------|------------------------------------------------|
+    | NVIDIA GPU | BAAI/bge-base-en-v1.5                          |
+    | macOS      | sentence-transformers/all-MiniLM-L6-v2         |
+    | Other/CPU  | sentence-transformers/all-MiniLM-L6-v2         |
+    """
+
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            return "BAAI/bge-base-en-v1.5"
+    except ImportError:
+        pass
+
+    # macOS (MPS or CPU) and all other platforms: lightweight model
+    return "sentence-transformers/all-MiniLM-L6-v2"
+
+
 def _normalize_path(path: str) -> str:
     """Return absolute path string for consistent keys."""
     if not path:
@@ -260,11 +282,12 @@ def create_parser(self) -> argparse.ArgumentParser:
             choices=["hnsw", "diskann", "ivf"],
             help="Backend to use (default: hnsw)",
         )
+        _default_model = _default_embedding_model()
         build_parser.add_argument(
             "--embedding-model",
             type=str,
-            default="facebook/contriever",
-            help="Embedding model (default: facebook/contriever)",
+            default=_default_model,
+            help=f"Embedding model (default: {_default_model})",
         )
         build_parser.add_argument(
             "--embedding-mode",
@@ -2432,8 +2455,15 @@ async def build_index(self, args):
                 is_compact = meta.get(
                     "is_compact", meta.get("backend_kwargs", {}).get("is_compact", True)
                 )
+
+                # Normalize model names for comparison — e.g. "all-MiniLM-L6-v2"
+                # should match "sentence-transformers/all-MiniLM-L6-v2"
+                def _normalize_model_name(name: str) -> str:
+                    return name.rsplit("/", 1)[-1] if name else name
+
                 same_embedding = (
-                    meta.get("embedding_model") == args.embedding_model
+                    _normalize_model_name(meta.get("embedding_model", ""))
+                    == _normalize_model_name(args.embedding_model)
                     and meta.get("embedding_mode") == args.embedding_mode
                 )
 
@@ -2519,6 +2549,11 @@ async def build_index(self, args):
 
                 else:
                     self._log_rebuild_reason(meta, args, new_paths, removed_paths, modified_paths)
+                    # Force full reload — the partial all_texts from incremental
+                    # path only contains changed files, not the full corpus.
+                    all_texts = self.load_documents(
+                        docs_paths, args.file_types, include_hidden=args.include_hidden, args=args
+                    )
 
         # Full rebuild: load documents if not already loaded (first build or force)
         try:
@@ -3135,7 +3170,7 @@ async def search_documents(self, args):
             json_results = [
                 {
                     "id": r.id,
-                    "score": r.score,
+                    "score": float(r.score),
                     "text": r.text,
                     "metadata": r.metadata,
                 }