[Bug]: `delete_collection` leaves list-valued metadata behind, which reappears on records written to the next collection

[sjmurdoch]

What happened?

What happened

After delete_collection, a newly created collection reports count() == 0, but the first record added to it silently inherits the list-valued metadata of the deleted collection's first record.

Scalar metadata is cleaned up correctly. Only list values leak. The leak also crosses collection names — the replacement collection does not have to reuse the deleted one's name — and it survives a process restart, so it is on-disk state rather than an in-process cache.

This is not merely wasted space: it returns metadata that was never written to the new record, so where filters match documents on keys their content does not have.

Minimal reproduction

import chromadb

client = chromadb.PersistentClient(path="/tmp/chroma-ghost")

collection = client.get_or_create_collection("demo")
collection.add(ids=["id-1"], embeddings=[[0.1, 0.2, 0.3]], metadatas=[{"listy": ["ghost"]}])
client.delete_collection("demo")

collection = client.get_or_create_collection("demo")
assert collection.count() == 0                      # passes

collection.add(ids=["id-1"], embeddings=[[0.1, 0.2, 0.3]], metadatas=[{"scalar": "new"}])
print(collection.get(ids=["id-1"], include=["metadatas"])["metadatas"][0])

Expected: {'scalar': 'new'}

Actual: {'listy': ['ghost'], 'scalar': 'new'}

Run against a directory that does not already exist. No embedding model is required — embeddings are passed explicitly, so chromadb is the only dependency.

Scope

Case	Result
list value, collection recreated under the same name	leaks
list value, replacement collection has a different name	leaks
scalar value instead of a list	clean
collection.delete(ids=[...]) within a live collection, then re-add	clean
fresh directory	clean

The fourth row is worth noting: per-record deletion does clean up array metadata correctly. The defect is specific to delete_collection.

Where it comes from

Metadata is stored in two tables. After the reproduction above, with one record in a supposedly fresh collection:

embedding_metadata:       [(1, 'scalar', 'new')]
embedding_metadata_array: [(1, 'listy', 'ghost')]

delete_collection removes the rows in embedding_metadata but leaves those in embedding_metadata_array. The new collection's first record is assigned internal id 1 again, so it adopts the orphaned array rows. That also explains why a differently-named collection is affected: the orphans are keyed by the reused internal id, not by collection.

Why it matters

Found in a local RAG tool that stores Obsidian note metadata — tags, wikilinks, heading paths — all list-valued. Rebuilding the index with delete_collection + re-ingest left 1,647 of 38,622 chunks (in 531 of 8,390 files) carrying tags and links belonging to notes that had previously occupied the same chunk id. A tag filter returned six notes where only two genuinely carried the tag.

• count() reports 0, so the collection looks fresh.
• Documents and scalar metadata are correct, so a text- or hash-based integrity check passes.
• Each rebuild makes it worse, because every write resurrects whatever that id previously accumulated.

The only reliable workaround we found is to delete the persist directory rather than the collection.

Reproduction script

poc.py runs the minimal case, the supporting matrix, the two-process check, and prints the two SQLite tables:

python poc.py            # minimal case, then the matrix and table dump
python poc.py minimal    # just the reproduction above
python poc.py write      # phase 1, then run `reuse` in a separate process
python poc.py reuse      # phase 2 — shows the state is on disk

Related but distinct

#3793 and similar reports concern data left on disk after deletion — a space issue. This one is a correctness issue: the orphaned rows are re-attached to new records and returned in query results.

Versions

• chromadb 1.5.9
• Python 3.13
• macOS (arm64)
• PersistentClient, default (local) configuration

Relevant log output

[Xsidz]

Working on this in PR #7595.

[dineshyadav03]

Confirmed the repro against the PR — the on-disk state matches: embedding_metadata rows get cleared on delete_collection but embedding_metadata_array doesn't, so SQLite's reused autoincrement id on embeddings reattaches orphaned array rows to whatever record lands on that id next. Worth flagging that this is more than a leftover-bytes bug: a where filter on a brand-new collection can silently match on metadata keys the current record never wrote, and since the leak is keyed by internal id rather than collection name/id, it crosses collection boundaries. In any multi-tenant setup that reuses collection names/ids across tenants, that's effectively cross-tenant data leakage through query results, not just wasted disk space.

Glad to see PR #7595 targets both the Rust sysdb path and the Python SqliteMetadataSegment, matching where scalar cleanup already happens correctly. One thing I didn't see in the PR's test plan: is there an existing metadata-store integration test suite that exercises delete/recreate semantics? If so, it'd be worth adding a list-valued-field case there specifically, since a scalar-only fixture would pass today and still miss this class of regression.