fix(mesh): publish the estate so every transport can report it (#2309)

The peer sync loop is started only by _serve_http, and the estate it
builds (_PEER_SYNC_STATE, _KNOWN_PROFILES) is process memory. But
mempalace_mesh_peers ships in every transport, including the stdio
servers agents connect through, and those processes never run a sync
round.

So they answered from a permanently empty estate: every peer reduced to
a bare name and url with no reachable, last_success_at,
remote_version_vector or profile; origin_profiles holding only this
node; and configured peers reported as unnamed_origins -- "known only
transitively" -- because their replica_id is learned during a sync and
nothing else supplies it. The hub next door had all of it.

The sync loop now publishes the estate to mesh_state.json in the
per-palace server state directory after every round, alongside the
token and serverinfo that already use that directory for exactly this
"hub records something other local processes read" purpose. 0600, and
written to a temp name then renamed so a reader in another process
never observes a half-serialized estate.

_mesh_peers_payload merges the published estate underneath any
in-process state, per peer, so the process that actually syncs keeps
reporting its own fresher observation and every other process reports
the hub's instead of nothing.

The new estate_source field says where the reading came from:
in_process, published_at, and whether the publishing hub is still
alive. A crashed hub leaves a last-known-good estate, which is worth
showing -- "last seen as" beats a blank node -- but must not be read as
live.

peers.json tokens are not in the estate and never reach the file; both
are asserted.

Author: igorls

CHANGELOG.md

@@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 ### Bug Fixes
 
 - **A long-running MCP server no longer grows by ~440 MB per collection open.** `ChromaBackend._client` keys its client cache on `chroma.sqlite3`'s inode and mtime to detect writes by another process, but constructing a `chromadb.PersistentClient` writes to that file itself, and so do the collection opens that follow — so the stamp taken at construction time was already stale by the time the next open compared against it. Every search opens `mempalace_drawers` and then `mempalace_closets`, and the first open moved the mtime the second one checked, so the cache missed on essentially every request: each search rebuilt the client and reloaded both HNSW segments, and the displaced client was dropped from the dict without being closed, so its native index memory was never returned. On a palace of ~165k vectors that was ~650 MB of resident growth per search; a server left running for an afternoon reached 3.9 GB. The freshness stat is now re-baselined once the backend's own operation finishes — including writes through `ChromaCollection`, so the file-a-drawer-then-search cycle stops reloading the index too — which makes the recorded value mean "`chroma.sqlite3` as this backend last left it", and a genuine external change still rebuilds. The rebuild path also closes the client it replaces. Measured over eight searches on the same palace: 951 MB → 3522 MB before, 940 MB → 965 MB after. An external write landing *while* one of our own operations is in flight is absorbed into the new stamp and picked up on the next change; mtime cannot distinguish writers, and `PRAGMA data_version` does not help because chromadb's own connection reads as foreign to a probe connection. (#2307)
+- **`mempalace_mesh_peers` answers with the real estate from every transport, not just the hub.** The peer sync loop is started only by `_serve_http`, and the estate it builds (`_PEER_SYNC_STATE`, `_KNOWN_PROFILES`) lives in process memory — but the tool that reports it ships in every transport, including the stdio servers agents actually connect through. Those processes never run a sync round, so they answered from a permanently empty estate: each peer reduced to a bare `name` and `url` with no `reachable`, `last_success_at`, `remote_version_vector` or `profile`, `origin_profiles` holding only this node, and configured peers misreported as `unnamed_origins` because their `replica_id` was never learned — all while the hub next door had the complete picture. The sync loop now publishes the estate to `mesh_state.json` in the per-palace server state directory (0600, write-and-rename) after every round, and `_mesh_peers_payload` merges it underneath any in-process state, so the syncing process still trusts its own fresher reading. A new `estate_source` field says whether the status was observed in this process or published by the hub, when it was published, and whether that hub is still alive — a crashed hub leaves a last-known-good estate that is shown but never read as live. peers.json tokens never reach the file. (#2309)
 - **`sweep` books a failed `stat` as a failure again.** The non-regular-file gate added in 3.7.1 reads `stat.S_ISREG(f.stat().st_mode)` inside a `try`, and its `except OSError` printed `SKIP` and moved on. A dangling symlink, a symlink loop and a file unlinked between `rglob` and the gate all raise there, and before the gate existed every one of them reached `sweep()` and was booked in `failures` — so `sweep` went from reporting a transcript it could not read to reporting success. A failed probe is now an error, not a benign file type: it is logged, printed as `WARNING`, and appended to `failures`, while a probe that succeeds and says "not regular" still skips silently. (#2221)
 - **`mempalace init` no longer tracebacks on a directory it cannot enter.** `_parse_gradle`'s `is_file()` gate sat in front of the `try` that the parser's own `except OSError` provides, so a manifest under a directory with `r` but no `x` raised `PermissionError` out of a call that used to answer "no manifest name". The gate moved inside that `try`, and `_collect_manifest_names` stats through `os.path.isfile`, which reports rather than raises. (#2221)
 - **`split` no longer blocks on a FIFO at its own output name, nor writes through a broken link.** The type gate in `main` covers the files the glob listed; `split_file` builds its output names itself, so a pre-existing named pipe at one of them wedged `write_text` in the kernel waiting for a reader. The check asks about the link itself rather than its target, because a dangling symlink at an output name reads as "nothing there" and `write_text` would create the target — landing a chunk outside the output directory. Output names that are anything but a regular file are now skipped with a `SKIP` line. (#2221)

mempalace/mcp_server.py

@@ -4854,7 +4854,7 @@ def tool_patch_submit(
         "handler": tool_graph_stats,
     },
     "mempalace_mesh_peers": {
-        "description": "Mesh estate snapshot (RFC 004): this replica's identity, version vector and node profile; each configured peer's reachability, last sync outcome, remote version vector and advertised profile; origins known only transitively; and origin_profiles keyed by replica_id. Exactly the GET /sync/peers payload — tokens are never included.",
+        "description": "Mesh estate snapshot (RFC 004): this replica's identity, version vector and node profile; each configured peer's reachability, last sync outcome, remote version vector and advertised profile; origins known only transitively; origin_profiles keyed by replica_id; and estate_source saying whether the peer status was observed in this process or published by the palace's hub (with published_at and whether that hub is still alive). Exactly the GET /sync/peers payload — tokens are never included.",
         "input_schema": {"type": "object", "properties": {}},
         "handler": tool_mesh_peers,
     },
@@ -7234,17 +7234,60 @@ def _merge_known_profiles(profiles: dict) -> None:
         _KNOWN_PROFILES[origin] = profile
 
 
-def _known_profiles_snapshot() -> dict:
-    """Every origin profile this node can vouch for having seen — learned
-    ones first, own fresh self-profile last so it always wins for self."""
-    snapshot = dict(_KNOWN_PROFILES)
+def _known_profiles_snapshot(published: dict = None) -> dict:
+    """Every origin profile this node can vouch for having seen.
+
+    Published profiles first (the hub's, when this process is not the one
+    syncing), then any learned in this process, then our own fresh
+    self-profile last so it always wins for self.
+    """
+    if published is None:
+        published = _published_mesh_state()
+    snapshot = dict(published.get("profiles") or {})
+    snapshot.update(_KNOWN_PROFILES)
     try:
         snapshot[_call_logstream(lambda ls: ls.replica_id)] = _node_profile()
     except Exception:
         logger.debug("node profile: self profile unavailable", exc_info=True)
     return snapshot
 
 
+def _publish_mesh_state(palace_path: str) -> None:
+    """Write this process's estate where other local processes can read it.
+
+    The sync loop runs only in the HTTP transport, so without this the stdio
+    MCP servers agents connect through answer ``mempalace_mesh_peers`` from a
+    permanently empty ``_PEER_SYNC_STATE`` -- peers with a name and a url and
+    nothing else, and ``origin_profiles`` holding only this node. Never fatal:
+    a publish failure costs other processes their estate view, not this
+    process's convergence.
+    """
+    from . import server_registry
+
+    try:
+        server_registry.write_mesh_state(
+            palace_path,
+            peers=dict(_PEER_SYNC_STATE),
+            profiles=dict(_KNOWN_PROFILES),
+        )
+    except Exception:
+        logger.debug("mesh state publish failed", exc_info=True)
+
+
+def _published_mesh_state() -> dict:
+    """Read the estate published by this palace's hub, if any."""
+    from . import server_registry
+
+    palace_path = getattr(_config, "palace_path", None)
+    if not palace_path:
+        return {"peers": {}, "profiles": {}, "written_at": None, "writer_alive": False}
+    try:
+        return server_registry.read_mesh_state(palace_path)
+    except Exception:
+        logger.debug("mesh state read failed", exc_info=True)
+        return {"peers": {}, "profiles": {}, "written_at": None, "writer_alive": False}
+
+
 def _record_peer_sync(stats: dict) -> None:
     """Fold one peer's round outcome into the estate state."""
     name = stats.get("peer_name") or stats.get("peer_url") or "?"
@@ -7298,11 +7341,18 @@ def _mesh_peers_payload() -> dict:
         configured = load_peers(getattr(_config, "palace_path", None) or "")
     except (ValueError, TypeError):
         configured = []
+    # The peer sync loop lives in the HTTP transport, so in every other
+    # process _PEER_SYNC_STATE is empty and the only honest source is what
+    # the hub published. Prefer our own state when we have it (this process
+    # is the one syncing, so it is fresher than anything on disk) and fall
+    # back to the published estate per peer.
+    published = _published_mesh_state()
+    published_peers = published.get("peers") or {}
     named_origins = {replica_id}
     peers = []
     for peer in configured:
         name = peer.get("name") or peer["url"]
-        state = dict(_PEER_SYNC_STATE.get(name) or {})
+        state = dict(_PEER_SYNC_STATE.get(name) or published_peers.get(name) or {})
         state.pop("url", None)  # peers.json is authoritative for the url
         if state.get("replica_id"):
             named_origins.add(state["replica_id"])
@@ -7320,8 +7370,17 @@ def _mesh_peers_payload() -> dict:
         "unnamed_origins": sorted(set(local_vector) - named_origins),
         # Every self-described profile known here, keyed by replica_id —
         # including profiles of unnamed origins relayed through carriers.
-        "origin_profiles": _known_profiles_snapshot(),
+        "origin_profiles": _known_profiles_snapshot(published),
         "sync_interval_s": _peer_sync_interval_s(),
+        # Where the peer status above came from. in_process means this
+        # process runs the sync loop; otherwise the estate was published by
+        # the hub at published_at, and writer_alive says whether that hub is
+        # still running (a dead hub leaves a last-known-good reading).
+        "estate_source": {
+            "in_process": bool(_PEER_SYNC_STATE),
+            "published_at": published.get("written_at"),
+            "writer_alive": published.get("writer_alive", False),
+        },
     }
 
 
@@ -7373,6 +7432,9 @@ def _loop():
                             stats["pulled_events"],
                             stats["pulled_artifacts"],
                         )
+                # Publish once per round, not per peer: the estate is only
+                # coherent after every configured peer has been attempted.
+                _publish_mesh_state(palace_path)
                 malformed_logged = False
             except ValueError as exc:
                 if not malformed_logged:

mempalace/server_registry.py

@@ -28,6 +28,7 @@
 import json
 import logging
 import os
+import time
 from pathlib import Path
 
 logger = logging.getLogger(__name__)
@@ -41,6 +42,10 @@ def _canonical(palace_path: str) -> str:
     return os.path.abspath(os.path.realpath(os.path.expanduser(palace_path)))
 
 
+def _utc_now() -> str:
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+
+
 def server_state_dir(palace_path: str) -> Path:
     """Per-palace state directory shared by the token and the serverinfo.
 
@@ -88,6 +93,81 @@ def write_serverinfo(palace_path: str, *, host: str, port: int, scheme: str, rea
     return path
 
 
+def mesh_state_path(palace_path: str) -> Path:
+    return server_state_dir(palace_path) / "mesh_state.json"
+
+
+def write_mesh_state(palace_path: str, *, peers: dict, profiles: dict) -> Path:
+    """Publish the hub's mesh estate so other local processes can read it.
+
+    The estate — which peers answered last round, their version vectors and
+    advertised profiles — is built by the peer sync loop, and that loop only
+    runs in the HTTP transport. Every other process for this palace (the
+    stdio MCP servers agents actually connect through, the CLI) has the same
+    ``mempalace_mesh_peers`` tool and an empty in-memory estate behind it, so
+    without this file they answer "two peers, no status at all" while the hub
+    next door knows the whole picture.
+
+    0600 like the token and the serverinfo. The contents are not secret —
+    peers.json tokens never reach the estate — but the directory convention
+    is "private to the user".
+    """
+    path = mesh_state_path(palace_path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    try:
+        os.chmod(str(path.parent), 0o700)
+    except OSError:
+        pass
+    payload = {
+        "pid": os.getpid(),
+        "written_at": _utc_now(),
+        "peers": peers,
+        "profiles": profiles,
+    }
+    # Write-and-rename: readers in other processes must never observe a
+    # half-serialized estate, and this file is rewritten every sync round.
+    tmp = path.with_name(path.name + f".{os.getpid()}.tmp")
+    fd = os.open(str(tmp), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            json.dump(payload, fh)
+            fh.write("\n")
+        os.replace(str(tmp), str(path))
+    except BaseException:
+        try:
+            os.unlink(str(tmp))
+        except OSError:
+            pass
+        raise
+    return path
+
+
+def read_mesh_state(palace_path: str) -> dict:
+    """Return the published estate for this palace.
+
+    Always a dict with ``peers``/``profiles`` mappings so callers can merge
+    without None-checks; empty when no hub has published yet or the file is
+    unreadable. ``writer_alive`` reports whether the publishing process is
+    still running — a crashed hub leaves a last-known-good estate that is
+    worth showing but must not be read as live.
+    """
+    empty = {"peers": {}, "profiles": {}, "written_at": None, "writer_alive": False}
+    try:
+        state = json.loads(mesh_state_path(palace_path).read_text(encoding="utf-8"))
+    except (OSError, ValueError):
+        return empty
+    if not isinstance(state, dict):
+        return empty
+    peers = state.get("peers")
+    profiles = state.get("profiles")
+    return {
+        "peers": peers if isinstance(peers, dict) else {},
+        "profiles": profiles if isinstance(profiles, dict) else {},
+        "written_at": state.get("written_at"),
+        "writer_alive": _pid_alive(state.get("pid")),
+    }
+
+
 def clear_serverinfo(palace_path: str) -> None:
     """Remove this process's serverinfo record, if it is still ours.