feat: improve embedding throughput on cpu (#78)

* feat: improve embedding throughput on cpu

* docs: add cpu-only testing note

* fix: address embedding review feedback

Author: gmickel

CHANGELOG.md

@@ -9,6 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+### Fixed
+
+## [0.28.0] - 2026-03-23
+
+### Fixed
+
+- `gno embed` now requests full embedding threads from `node-llama-cpp` and uses an adaptive embedding-context pool on CPU-only machines, with graceful fallback when extra contexts cannot be created. Thanks @riyadist for the report and repro details.
+
 ## [0.27.3] - 2026-03-23
 
 ### Fixed
@@ -853,7 +861,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 | 0.4.0   | 2026-01-01 | Web UI and REST API                       |
 | 0.1.0   | 2025-12-30 | Initial release with full search pipeline |
 
-[Unreleased]: https://github.com/gmickel/gno/compare/v0.27.3...HEAD
+[Unreleased]: https://github.com/gmickel/gno/compare/v0.28.0...HEAD
+[0.28.0]: https://github.com/gmickel/gno/compare/v0.27.3...v0.28.0
 [0.27.3]: https://github.com/gmickel/gno/compare/v0.27.2...v0.27.3
 [0.27.2]: https://github.com/gmickel/gno/compare/v0.27.1...v0.27.2
 [0.27.1]: https://github.com/gmickel/gno/compare/v0.27.0...v0.27.1

docs/CLI.md

@@ -52,6 +52,20 @@ All commands accept:
 
 **Offline mode**: Use `--offline` or set `HF_HUB_OFFLINE=1` to prevent auto-downloading models. Set `GNO_NO_AUTO_DOWNLOAD=1` to disable auto-download while still allowing explicit `gno models pull`.
 
+**Force CPU-only for testing**: Set `NODE_LLAMA_CPP_GPU=false` on the `gno`
+process to disable Metal/CUDA/Vulkan and force the CPU backend:
+
+```bash
+NODE_LLAMA_CPP_GPU=false gno doctor --json
+NODE_LLAMA_CPP_GPU=false gno embed --yes
+```
+
+Accepted values from `node-llama-cpp`: `false`, `off`, `none`, `disable`,
+`disabled`.
+
+> **Note:** the first CPU-only run may build or download a separate CPU backend
+> if you only have GPU-backed binaries cached. Use the second run for timing.
+
 **Output format flags** (`--json`, `--files`, `--csv`, `--md`, `--xml`) are per-command.
 See [spec/cli.md](../spec/cli.md#output-format-support-matrix) for which commands support which formats.
 
@@ -373,6 +387,10 @@ Options:
 
 Generate embeddings for indexed chunks.
 
+On CPU-only machines, GNO uses a small adaptive pool of embedding contexts to
+keep more cores busy. If RAM is tight, it automatically falls back to fewer
+contexts.
+
 ```bash
 gno embed
 gno embed notes

docs/TROUBLESHOOTING.md

@@ -208,6 +208,22 @@ gno models pull --embed
 gno models pull --all
 ```
 
+### Force CPU-only for testing
+
+To disable Metal/CUDA/Vulkan and force `node-llama-cpp` onto the CPU backend
+for a repro or benchmark:
+
+```bash
+NODE_LLAMA_CPP_GPU=false gno doctor --json
+NODE_LLAMA_CPP_GPU=false gno embed --yes
+```
+
+Accepted values: `false`, `off`, `none`, `disable`, `disabled`.
+
+If your machine only has GPU-backed `node-llama-cpp` binaries cached, the first
+CPU-only run may build or download a separate CPU backend. For throughput
+measurements, ignore that first run and time the second.
+
 ## Model Issues
 
 ### Models Fail to Download

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gmickel/gno",
-  "version": "0.27.3",
+  "version": "0.28.0",
   "description": "Local semantic search for your documents. Index Markdown, PDF, and Office files with hybrid BM25 + vector search.",
   "keywords": [
     "embeddings",

spec/cli.md

@@ -411,6 +411,10 @@ With `--verbose`, embedding errors during the embed phase are logged to stderr (
 
 Generate embeddings for chunks without vectors.
 
+On CPU-only machines, implementations may use multiple embedding contexts
+internally to improve throughput, and may fall back to fewer contexts if
+memory pressure prevents creating the full pool.
+
 **Synopsis:**
 
 ```bash