fix(chroma): flag stale HNSW ghost surplus - #2262
Draft
fallenmi wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Refs #2176.
After repeated or bulk deletes, Chroma can retain substantially more persisted HNSW elements than live SQLite embeddings. The reported case had 1,606 SQLite rows and 9,974 HNSW elements (
divergence=-8,368), but the capacity probe classified every negative divergence asOKand explicitly called the surplus safe.This PR fixes the diagnostic and recovery path:
status="stale"andrepair_recommended=true;divergedflag remainsfalsefor this direction, so the native-crash fence reserved for an undersized HNSW index does not disable vector reads or writes;OK;repair-statusprintsSTALEand recommends rebuilding from SQLite, including when all live drawer rows were deleted but ghost HNSW elements remain;repair_recommendedfield.Scope
This is deliberately a detection and recovery-guidance fix. It does not compact or prevent Chroma's deleted HNSW elements, and it does not add cumulative-delete accounting or a hint to every
delete_by_sourceresponse. Those can be handled separately without coupling a potentially expensive repair to the write path.Safety
diverged=truecurrently activates the #1222 fail-closed path: BM25-only search, vector-write refusal, and several preflight aborts. The surplus reported in #2176 degraded ranking but was not shown to create the undersized-index native-crash risk, so the newstalestate recommends repair while intentionally leaving vectors enabled.How to test
Local results on current
develop:test_init_filters_sys_path_from_leaked_pythonpathsubprocess assertion excluded;git diff --checkpassed.Regression coverage pins the exact 1,606/9,974 report, explicit and legacy threshold boundaries, zero-live-row ghost state, repair output, vector-fence behavior, and warning deduplication.