mempalace init replaces a known_entities.json it could not read, and ends in a traceback when it is not UTF-8 #2358

What happened?

mempalace init replaces a known_entities.json it could not read with the entities of that one run and reports it as an update. When the file is not valid UTF-8 it does not get that far: the run ends in a traceback.

add_to_known_entities reads the registry behind Path.exists() and falls back to an empty dict (miner.py:917-924 on develop), then writes the merged dict over the whole file at :980:

existing: dict = {}
if registry_path.exists():
    try:
        loaded = _json.loads(registry_path.read_text(encoding="utf-8"))
        if isinstance(loaded, dict):
            existing = loaded
    except (_json.JSONDecodeError, OSError):
        existing = {}

Nothing between the fallback and the write asks whether the registry was read. cli.py:460 is the only caller, and it prints Registry updated: <path> afterwards.

Measured through the CLI on a project with a git history and a pyproject.toml, so init has entities to record. For scale: a clean init on a shallow clone of this repository, 126 commit authors, writes a 366-byte registry holding people (15 names, the cap) and projects (1).

The first two lose every name and wing the registry held, for every project it was built from, and say the opposite. The third is not a data loss but a command that does not finish: UnicodeDecodeError is neither JSONDecodeError nor OSError, so it passes the except and out through cli.py. config.py:379 catches that exception explicitly, in the same package, for the same reason.

None of the three needs an interrupted write. The registry is not documented anywhere, but it sits next to the config.json that is, and it holds names a scan chose rather than names anyone typed: 15 of those 126 authors, bots and duplicates included. Editing it by hand is how they get cleaned up, and one editor saving it in the machine's legacy encoding is the cp1251 row. The write is not atomic either, since write_text builds the document in memory and then truncates the file, so an interrupted run or a full disk leaves the same kinds of file.

Widening that except to catch UnicodeDecodeError is not the fix it looks like. Measured on a 94-byte registry, it stops the traceback and turns the run into a silent replacement: the file becomes 38 bytes holding that run's one name, and nothing keeps what was there. The crash at least leaves the bytes alone.

What did you expect?

FileNotFoundError is the one outcome that establishes there is no registry here. Everything else says this process did not read the one that is there, which is a reason to keep it rather than to write over it: rename it aside and print where it went, or leave it alone and say so. A file that is not valid UTF-8 belongs in the same branch rather than in a traceback.

And this file already has an atomic writer in this package. migrate._apply_topics_by_wing_renames writes known_entities.json through tempfile.mkstemp and os.replace (migrate.py:529-534), and seven other state files are published by a rename as well: hallways.py:146, entity_registry.py:339, palace_graph.py:546, repair.py:114, replica.py:63, server_registry.py:135 and logstream.py:398. The registry's own merge path is the one that truncates.

Note that two existing tests pin the current behaviour rather than an accident: test_malformed_existing_registry_starts_fresh and test_non_dict_existing_registry_starts_fresh (tests/test_known_entities_registry.py:127 and :134) assert that the file is replaced. Keeping the old bytes beside the new registry satisfies both as written, since what the new registry holds does not change.

How to reproduce:

1. mempalace init <project-a> and confirm some entities, so ~/.mempalace/known_entities.json holds them.
2. Make it unreadable the way an editor would: printf '["a","b"]' > ~/.mempalace/known_entities.json, or re-save it in a legacy encoding, or truncate it: python -c "import os; p=os.path.expanduser('~/.mempalace/known_entities.json'); os.truncate(p, os.path.getsize(p)//2)".
3. mempalace init <project-b> --yes --no-llm and confirm its entities.
4. Read the registry. For the array and the truncated file it holds project B's categories only, project A's entities and every topics_by_wing entry are gone, and step 3 printed Registry updated. For the legacy encoding, step 3 ended in a UnicodeDecodeError traceback and wrote no config.

Environment:

• OS: Linux (WSL2)
• Python version: 3.12.3
• MemPal version: develop @ 4bc0c43

Notes

topics_by_wing is the signal palace_graph reads for topic tunnels at mine time, and _load_known_entities (miner.py:1061) is what tags drawer metadata for entity-filtered search, so a lost registry is not only a lost file: the next mine files drawers without the entities and tunnels the old one would have produced, and nothing about that mine looks unusual. mempalace mine itself does not fail on any of the three files.

This is the overwrite half of the shape #2320 reports for deletions, and that report names this function as one of the two places outside its scope. The other, MempalaceConfig, is #2356.

Four more places fold a read that did not conclude into an empty default and write that default back over the same file: palace_graph._load_tunnels (:490-499), hallways._load_hallways (:98-104), EntityRegistry.load (entity_registry.py:308-316) and write_embedder_sidecar (backends/_sidecar.py:53-68), whose docstring says it preserves the other collections' entries. The first two are reached by a plain mempalace mine and both write atomically, so what they drop is dropped durably. Each is its own report, none filed yet.

CLAUDE.md states the rule this breaks: "Never destroy existing data to rebuild. A crash mid-operation must leave the existing palace untouched."