Deleted documents remain in embeddings_queue on 1.5.9 with automatically_purge=true (reproduces closed #3793)

[rutvikbuilds]

Summary

#3793 — "Document deletion leaves plain text and embeddings in db" — was closed as completed on 2025-06-30 with #4884 named as the fix. A commenter reported afterwards that it still reproduced on 1.0.15 and the thread went quiet.

It still reproduces on 1.5.9 (current release), on a store created fresh by 1.5.9, with automatically_purge enabled — which is the default for a new store. embeddings_queue is never trimmed: it grows monotonically and retains the document text and embedding of every deleted record.

Reproduction

chromadb==1.5.9, Python 3.12, macOS arm64, PersistentClient.

import hashlib, os, shutil, sqlite3
import chromadb

DB = os.path.abspath("chroma-persist")
if os.path.exists(DB):
    shutil.rmtree(DB)

def embed(texts):
    return [[(hashlib.sha256(t.encode()).digest()[i % 32] / 255.0) for i in range(32)]
            for t in texts]

def wal(label):
    con = sqlite3.connect(os.path.join(DB, "chroma.sqlite3"))
    n = con.execute("SELECT count(*) FROM embeddings_queue").fetchone()[0]
    cfg = con.execute("SELECT * FROM embeddings_queue_config").fetchall()
    raw = open(os.path.join(DB, "chroma.sqlite3"), "rb").read()
    print(f"{label:26} queue={n:<5} subject on disk: {b'Dana Whitfield' in raw}  {cfg}")

SUBJECT = "Dana Whitfield lives at 42 Alder Row, dana.whitfield@example.com"

client = chromadb.PersistentClient(path=DB)
col = client.create_collection("memories")
col.add(ids=["m1", "m2", "m3"],
        documents=[SUBJECT, "Marcus prefers dark mode", "Kafka runs on 9092"],
        embeddings=embed([SUBJECT, "a", "b"]))
wal("after add 3")

col.delete(ids=["m1", "m2"])
wal("after delete 2")

del col, client
chromadb.api.client.SharedSystemClient.clear_system_cache()
client = chromadb.PersistentClient(path=DB)
col = client.get_collection("memories")
wal("after restart")

docs = [f"Filler record number {i}" for i in range(150)]
col.add(ids=[f"f{i}" for i in range(150)], documents=docs, embeddings=embed(docs))
wal("after 150 more writes")

Actual

after add 3                queue=3     subject on disk: True   [(1, '{"automatically_purge":true,...}')]
after delete 2             queue=5     subject on disk: True   [(1, '{"automatically_purge":true,...}')]
after restart              queue=5     subject on disk: True   [(1, '{"automatically_purge":true,...}')]
after 150 more writes      queue=155   subject on disk: True   [(1, '{"automatically_purge":true,...}')]

The queue only grows. Deleting two records adds two rows rather than removing two.

Expected

With automatically_purge true, entries for records that have been deleted and compacted should not remain in embeddings_queue indefinitely.

What the queue holds

seq=1 op=add     m1   128-byte vector   "Dana Whitfield lives at 42 Alder Row…"
seq=2 op=add     m2   128-byte vector   "Marcus prefers dark mode"
seq=3 op=add     m3   128-byte vector   "Kafka runs on 9092"
seq=4 op=delete  m1   (the add row above is not removed)
seq=5 op=delete  m2   (the add row above is not removed)

The embeddings table is pruned correctly and the API is consistent throughout — count() returns the right number and get() does not return the deleted ids. The residue is only visible below the API.

Other things tried

• chroma vacuum --path <dir> --force — reclaims some space (16 KiB, 4% on the 155-row store above) but does not purge the queue: embeddings_queue stays at 155 rows and the deleted records' text is still in the file afterwards.
• No client or collection method in 1.5.9 exposes a purge, compact or truncate operation.
• Two restarts and 150 further writes, in case a sync threshold gates it. No change.

Why it matters

Unbounded growth. This was the original reporter's complaint on #3793: a delete-then-reinsert workflow grows chroma.sqlite3 forever.

Deleted personal data persists in plaintext. For anyone using Chroma to hold personal data, delete() removes the record from the API while the document text and its embedding stay in the file. That is a data-retention exposure rather than a correctness bug — the API is telling the truth about its own view, and the data is still on disk. Embeddings are also not reliably non-identifying (Morris et al., Text Embeddings Reveal (Almost) As Much As Text), though here the plaintext is present anyway.

Not tested

• Client/server mode. The earlier commenter on [Bug]: Document deletion leaves plain text and embeddings in db #3793 reported the same behaviour on chromadb/chroma:1.0.15 in Docker, but that is their report, not mine.
• Whether a store that has never had automatically_purge written behaves differently.
• Non-macOS platforms.

Ask

Reopen #3793, or treat this as its continuation — happy to move the detail there instead if that is easier to track.

[jstar0]

I reproduced the queue-growth numbers on current main and compared them with the existing WAL purge tests (chromadb/test/db/test_log_purge.py and log_size_below_max in chromadb/test/property/invariants.py).

automatically_purge does not delete a row when the corresponding collection record is deleted. The WAL is append-only: delete() submits a DELETE operation (id only — no document text on that row). Purge then removes embeddings_queue rows with seq_id < min(segment max_seq_id) for that collection.

The HNSW segment only writes max_seq_id when it persists, which happens at hnsw:sync_threshold (default 1000). Until that persist, purge_log coalesces a missing vector-segment offset to -1 and deletes nothing. That is why a fresh 1.5.9 store with a few adds/deletes plus 150 more writes still shows a monotonically growing queue: 155 operations is still below the default persist threshold.

#4884 is also a different path (delete_collection removing embeddings/metadata rows), not this per-record WAL trim.

Two consequences of the current design:

1. Deleted document text stays in the earlier ADD WAL rows until the collection has persisted far enough for purge to run.
2. Even after those rows are deleted, a raw chroma.sqlite3 scan can still see the bytes until VACUUM (or chroma vacuum) rewrites the file.

I do not plan to change purge to run before the vector segment has persisted max_seq_id — that is the failure mode tracked in #2922. If maintainers want deleted payloads gone sooner, the safer options look like redacting vector/metadata on prior WAL rows for a deleted id after all in-process consumers have applied the DELETE, and/or documenting that auto-purge is tied to hnsw:sync_threshold plus vacuum. Happy to implement whichever of those you want.

[rutvikbuilds]

@jstar0 — thank you, and you are right. I reproduced your explanation and part of my original report is wrong. Correcting it here rather than leaving it standing.

What I got wrong

I wrote that the queue "only ever grows" and is "never trimmed". It is trimmed, exactly where you said — once the vector segment persists max_seq_id, which happens at hnsw:sync_threshold (default 1000). My test used 155 operations, which is simply below it.

Fresh 1.5.9 store, one add and one delete of a record containing the subject text, then filler writes in batches of 250:

after add 1 + delete it            queue=2      subject on disk: True
after 250 more writes              queue=252    subject on disk: True
after 500 more writes              queue=502    subject on disk: True
after 750 more writes              queue=752    subject on disk: True
after 1000 more writes             queue=1      subject on disk: True    <- purge fires
after 1250 more writes             queue=251    subject on disk: False
after restart                      queue=501    subject on disk: False

The purge is clearly working. The queue collapses from 752 to 1 at the threshold.

I should also correct the framing on #4884: I repeated the closing comment on #3793 that named it as the fix. You are right that it addresses delete_collection removing embeddings and metadata rows, which is a different path from per-record WAL trim. My issue implied that fix was supposed to cover this case and failed, and that is not accurate.

What I think still stands, narrower

Two things, and the second is weaker than the first:

1. On a store below the persist threshold, a deleted record's document text stays in the WAL indefinitely. That is not a growth problem, it is a retention one. A collection holding personal data that sees fewer than ~1000 operations retains the text of deleted records for as long as it stays quiet, and nothing in the public API forces the purge — chroma vacuum does not, and no client or collection method exposes one. For a low-traffic store answering a deletion request, "eventually, after a thousand more operations" is an awkward answer.

2. The bytes outliving the rows is ordinary SQLite behaviour and I would not treat it as a Chroma issue. In the run above the subject disappeared from the file between 1000 and 1250 writes, and that is page reuse overwriting freed space rather than anything erasing it. VACUUM is the real answer there.

On your proposed options

Redacting vector and metadata on prior WAL rows for a deleted id, once in-process consumers have applied the DELETE, is the one that addresses the retention concern without touching the purge ordering in #2922. That seems like the right shape to me, for whatever an outside reporter's opinion is worth.

Documenting that auto-purge is tied to hnsw:sync_threshold would also have saved me filing this in the first place, which is probably an argument for it.

Happy for maintainers to retitle or close this as appropriate — the original title overstates what I actually demonstrated.

[jstar0]

Thanks for the correction and the threshold-crossing numbers — they match what I measured.

I agree the remaining question is retention on a quiet collection, not unbounded growth. Redacting already-applied WAL rows for a deleted id (without changing the seq_id < min(max_seq_id) order) would need an explicit maintainer design pass so it does not replay into the #2922 class of loss. I will not change purge from this issue unless that direction lands.

Documenting that auto-purge waits on HNSW persist / hnsw:sync_threshold is a separate, safer follow-up if maintainers want it.