mempalace-issue-fts5-autoheal

[darkrhodes7]

FTS5 auto-heal is unreachable when the serving interpreter's SQLite cannot detect the fault

Version: 3.7.0 (also present in 3.5.0)
Platform: Windows 11, uv tool install mempalace
Backend: chroma, default minilm embedder
Palace: 322,848 drawers, chroma.sqlite3 ~1.95 GB, 325,451 embedding_fulltext_search_content rows

Summary

repair.maybe_autoheal_fts5_index only runs when repair.sqlite_integrity_errors
returns a non-empty list. That function asks the SQLite bundled with whichever
interpreter is running MemPalace. On this install that is SQLite 3.47.1, and
3.47.1 does not detect the FTS5 fault present in this palace, while SQLite
3.50.4 reports it on the same file through the same mode=ro URI.

The result is that a palace can carry a malformed FTS5 inverted index
indefinitely while every MemPalace-run check — the startup integrity probe,
repair's preflight, and the post-mine validation in
palace._validate_palace_fts5_after_mine — reports it clean. The auto-heal
added for exactly this condition can never fire.

Separately, and more surprising: a 'rebuild' issued by 3.47.1 does not clear
the fault, while the same statement issued by 3.50.4 does. So even if
detection were fixed, the heal would still no-op on this interpreter.

Evidence

Same file, same URI, both pragma forms, minutes apart:

SQLite	PRAGMA quick_check(1)	PRAGMA quick_check
3.50.4 (CPython 3.14.6)	malformed inverted index for FTS5 table main.embedding_fulltext_search	same
3.47.1 (CPython 3.12.9 — MemPalace's uv venv)	ok	ok

FTS5's own command, from 3.50.4:

INSERT INTO embedding_fulltext_search(embedding_fulltext_search) VALUES('integrity-check');
-- DatabaseError: database disk image is malformed

Table definition:

CREATE VIRTUAL TABLE embedding_fulltext_search USING fts5(string_value, tokenize='trigram')
-- embedding_fulltext_search_config: version = 4

Rebuild behaviour, on a byte copy of the live database so the result is
attributable to the rebuild alone:

'rebuild' issued by	duration	3.50.4 quick_check afterwards
3.47.1	76s	still malformed
3.50.4	85s	clean, and integrity-check passes

After rebuilding the live palace with 3.50.4 under mine_palace_lock,
quick_check is clean, integrity-check passes, embedding_fulltext_search_content
is unchanged at 325,451 rows, and MemPalace (still on 3.47.1) returns identical
search results with identical BM25 scores — so a 3.50.4-written index reads back
correctly on 3.47.1 at this scale.

What I could not reproduce

A synthetic fts5(body, tokenize='trigram') table of 4,000 rows round-trips
clean in both directions between 3.47.1 and 3.50.4, with and without an
explicit 'rebuild'. So this is not simply "3.50.4 rejects any trigram index
3.47.1 writes" — something about this palace's data or scale is required, and I
do not have a minimal reproduction. Candidates I did not chase: multi-segment
indexes at ~2 GB, and content predating the embedded-NUL sanitisation in the
3.6.0 line (#1716, #1872, #1878, #1927, #1928).

I am reporting it without that minimal case because the detection asymmetry
stands on its own and is straightforward to check.

Impact

Nothing is visibly broken while MemPalace stays on SQLite < 3.50: BM25 search
returns sensible scores, because the index is self-consistent for the build that
wrote it. The problem is latent and surfaces on interpreter upgrade — at which
point quick_check starts failing, repair aborts at the preflight with
"ABORT: SQLite-layer corruption detected before repair rebuild", and the operator
is told their palace is corrupt. On this palace the underlying content was intact
the whole time.

It also means the several auto-heal paths added in the 3.6.0/3.7.0 line are
silently inert on any install whose Python bundles an older SQLite, which
includes the documented uv tool install route.

Suggestions

1. Log the SQLite version alongside integrity verdicts. A "clean" from a
   build that cannot see this class of fault is not the same claim as a clean
   from one that can, and today they are indistinguishable in the output.
2. Consider a floor for the integrity check, or a note in
   docs/write-routing-policy.md that integrity results are only as good as the
   linked SQLite.
3. Worth confirming independently that FTS5 'rebuild' on 3.47.1 can leave a
   large trigram index in a state 3.50.4 rejects. If that holds generally it
   affects the heal path regardless of detection.

Happy to run further diagnostics against this palace — it reproduces reliably
here and I have a verified pre-change backup.

[mvalentsev]

Thanks for the version matrix — it made suggestion 1 straightforward, and #2254 implements it. That PR does not close this issue, so here is what held up and what did not.

Implemented. sqlite3.sqlite_version now accompanies every quick_check verdict: the repair abort banner, both --dry-run branches, the heal's outcome line, the post-recovery check, all three shapes of the MCP integrity payload, the tool-refusal payload, the integrity log line, and MineValidationError. That last one because you name the post-mine check, and the MCP mine tool and the daemon's job runner render the exception rather than the banner.

Two things fell out of reading that code. maybe_autoheal_fts5_index had two exits that declined silently, so an operator saw the ABORT banner with no sign a rebuild had been considered. And the MCP startup probe is skipped above 512 MB — never run on your ~1.95 GB palace — yet the payload still answered checked: true, ok: true. Both now report the absence of a verdict. That is your failure mode one layer up: a "clean" nobody produced.

Did not reproduce. Across 3,200 deliberately damaged FTS5 databases per build (trigram included), the same 2,975 came back dirty on 3.45.1, 3.47.1 and 3.51.2. Fingerprinting the dirty set confirms it is the same databases, not just the same count. They differed only in wording, and quick_check agreed with integrity-check 24 out of 24 times on each build. Your third observation did not reproduce either: a 'rebuild' left quick_check clean on all three, after damage to the index and to the content table alike.

Two caveats. I ran 3.45.1/3.47.1/3.51.2, not your 3.47.1/3.50.4 pair, and nowhere near 322k drawers. Your remaining candidates — multi-segment indexes at ~2 GB, content predating the NUL sanitisation — are exactly what a synthetic corpus cannot reach. So this says the asymmetry is not a general property of those builds. It says nothing against your report.

Worth knowing if you return to that palace. On 3.45.1 and 3.47.1, damaging the content table produces the same malformed inverted index wording as damaging the index. The classifier accepts it, so the heal rebuilds from the damaged text and reports success. Measured on 200 rows containing epsilon: after damaging 40 rows of %_content, MATCH 'epsilon' still returned 200, because the index held the original tokens. After the heal it returned 160, with "FTS5 index rebuilt from intact content; quick_check is clean" printed. A clean quick_check after a heal is not evidence the text survived. That belongs in its own issue.

If you can still run diagnostics against the pre-change backup, two things would move this furthest: whether PRAGMA integrity_check also splits between 3.47.1 and 3.50.4 on that file, and the row count of embedding_fulltext_search_data. The second would tell us whether multi-segment size is the trigger.

[darkrhodes7]

Hi Mikhail, Thanks for the detailed write-up and for #2254. I still had the pre-change backup, so I ran both of the checks you requested. In short: yes, integrity_check splits as well, and the difference is clear. The file is chroma.sqlite3 (1.95 GiB, 325,451 embeddings). This is the palace from the original report, kept as a backup from 2026-08-13 just before I removed a large wing from it. Since the live store is now down to a few hundred drawers, this backup is the only copy that preserves the state showing the problem. Everything below was run against that file in read-only mode (mode=ro&immutable=1) on the same machine, back to back. 1. Does integrity_check also split? Yes. SQLite 3.47.1 (CPython 3.13.1) - PRAGMA quick_check -> malformed inverted index for FTS5 table main.embedding_fulltext_search (103.3s) - PRAGMA integrity_check -> malformed inverted index for FTS5 table main.embedding_fulltext_search (28.4s) SQLite 3.50.4 (CPython 3.14.6) - PRAGMA quick_check -> ok (117.5s) - PRAGMA integrity_check -> ok (57.8s) Both instruments agree within a build and disagree between builds, which suggests this is not just a quick_check quirk. It is also the same wording your classifier already accepts, on a file that the newer SQLite considers completely clean. 2. Row counts - embedding_fulltext_search_data: 174,097 - embedding_fulltext_search_idx: 47,491 - embedding_fulltext_search_content: 325,451 - embeddings: 325,451 If multi-segment size is the trigger, the ratio of 174k data rows to 47k index rows is likely the key factor—that is a more complex segment structure than a synthetic corpus would typically build. Two details worth noting from my side: - I cannot rule out that this file was damaged at some point and that 3.50.4 simply does not detect that specific damage, rather than 3.47.1 misreporting it. The disagreement is stable and repeatable on this file. - The content of the palace predates a NUL-sanitisation change, which may still be a factor. The backup is saved, so let me know if you need any further diagnostics, such as segment-level details, fts5vocab output, or specific queries. Separately, the startup-probe finding is a great catch. A payload returning checked: true, ok: true for a check that was skipped over 512 MB is a critical finding. The content-damage measurement—where rows dropped from 200 to 160 while reporting "quick_check is clean"—is also a significant issue that I will be monitoring closely on my own store. Thanks again for investigating this report thoroughly. Best regards, Kuro

…

On Fri, Aug 14, 2026 at 12:07 PM Mikhail Valentsev ***@***.***> wrote: *mvalentsev* left a comment (MemPalace/mempalace#2240) <#2240 (comment)> Thanks for the version matrix — it made suggestion 1 straightforward, and #2254 <#2254> implements it. That PR does not close this issue, so here is what held up and what did not. *Implemented.* sqlite3.sqlite_version now accompanies every quick_check verdict: the repair abort banner, both --dry-run branches, the heal's outcome line, the post-recovery check, all three shapes of the MCP integrity payload, the tool-refusal payload, the integrity log line, and MineValidationError. That last one because you name the post-mine check, and the MCP mine tool and the daemon's job runner render the exception rather than the banner. Two things fell out of reading that code. maybe_autoheal_fts5_index had two exits that declined silently, so an operator saw the ABORT banner with no sign a rebuild had been considered. And the MCP startup probe is skipped above 512 MB — never run on your ~1.95 GB palace — yet the payload still answered checked: true, ok: true. Both now report the absence of a verdict. That is your failure mode one layer up: a "clean" nobody produced. *Did not reproduce.* Across 3,200 deliberately damaged FTS5 databases per build (trigram included), the same 2,975 came back dirty on 3.45.1, 3.47.1 and 3.51.2. Fingerprinting the dirty set confirms it is the same databases, not just the same count. They differed only in wording, and quick_check agreed with integrity-check 24 out of 24 times on each build. Your third observation did not reproduce either: a 'rebuild' left quick_check clean on all three, after damage to the index and to the content table alike. Two caveats. I ran 3.45.1/3.47.1/3.51.2, not your 3.47.1/3.50.4 pair, and nowhere near 322k drawers. Your remaining candidates — multi-segment indexes at ~2 GB, content predating the NUL sanitisation — are exactly what a synthetic corpus cannot reach. So this says the asymmetry is not a general property of those builds. It says nothing against your report. *Worth knowing if you return to that palace.* On 3.45.1 and 3.47.1, damaging the *content* table produces the same malformed inverted index wording as damaging the index. The classifier accepts it, so the heal rebuilds from the damaged text and reports success. Measured on 200 rows containing epsilon: after damaging 40 rows of %_content, MATCH 'epsilon' still returned 200, because the index held the original tokens. After the heal it returned 160, with "FTS5 index rebuilt from intact content; quick_check is clean" printed. A clean quick_check after a heal is not evidence the text survived. That belongs in its own issue. If you can still run diagnostics against the pre-change backup, two things would move this furthest: whether PRAGMA integrity_check also splits between 3.47.1 and 3.50.4 on that file, and the row count of embedding_fulltext_search_data. The second would tell us whether multi-segment size is the trigger. — Reply to this email directly, view it on GitHub <#2240?email_source=notifications&email_token=BTCMMY5FWXHYRW6AVYGEZPT5J5PONA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZG4YTIMJSHE32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5297141297>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BTCMMY3HRCNWJZ4RVX2VJF35J5PONAVCNFSNUABGKJSXA33TNF2G64TZHMYTEMBRGY2TMMRRGA5US43TOVSTWNJRGM3DGMZRGY4TNILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BTCMMY3PEAWJXAQUDETDTVD5J5PONA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZG4YTIMJSHE32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/BTCMMYZGFSUE5YC4BZWQLBL5J5PONA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZG4YTIMJSHE32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[mvalentsev]

Thanks, that settles it. Both pragmas split the same way on the same file, so it is not a quick_check quirk.

The row counts carry further than the segment question. embedding_fulltext_search_content and embeddings are both 325,451 on your palace, and that correspondence is what a check of the content table rests on: chroma writes each document into embedding_metadata under chroma:document, and into the FTS5 table at rowid = embeddings.id.

The content-damage measurement is now #2278, fixed in #2279. The heal checks the content table against embedding_metadata, restores what disagrees, and declines when it cannot conclude. lexical_search went from 12 to 0 across the old heal; across the new one it stays at 12.

Your other reading, that the file carries damage 3.50.4 does not detect, is still as plausible as mine.