Commit b46f18d
* feat(miner): add C# and .NET file extensions to READABLE_EXTENSIONS
Adds .cs, .csproj, .sln, .razor, and .cshtml so C#/.NET projects
are indexed by the project miner. .razor/.cshtml are analogous to
the already-supported .jsx/.tsx.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(miner): add support for Swift and Kotlin file extensions
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts".
- Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
* feat: add Pi agent JSONL session normalizer
Add _try_pi_jsonl parser for Pi agent session files stored at
~/.config/pi/agent/sessions/{encoded-cwd}/{timestamp}_{uuid}.jsonl.
Uses type "message" entries with role "user"/"assistant". Skips
toolResult messages, model_change, thinking_level_change, and other
operational events. Requires session header (type "session" with
"version" key) to avoid false positives.
Format documented at github.com/badlogic/pi-mono session.md and
verified via Context7. Sample data provided by tunnckoCore in #59.
Refs: #59
* feat: add Gemini CLI / AI Studio JSON session import support
Adds _try_gemini_json parser to normalize.py for three layouts:
1. Gemini API contents format (~/.gemini/sessions/*.json):
{"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}
2. Messages-wrapper variant:
{"messages": [{"role": "user", ...}, {"role": "model", ...}]}
3. Flat top-level list with role="model".
This complements the existing _try_gemini_jsonl parser (which handles
~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata
sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers
exported / Studio-saved transcripts.
## Review feedback addressed (PR #204)
bgauryy review:
- #1 Parser-precedence bug: _try_gemini_json runs *before*
_try_claude_ai_json so the {"messages":[..., role=model, ...]}
layout is no longer silently claimed by the Claude parser. The
Gemini parser's has_model_role guard prevents false-positives
against Claude / ChatGPT data.
- #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format
+ test_messages_wrapper_does_not_get_claimed_by_claude pin the
fix in place.
- #3 Test conflicts with current main: rebased onto develop;
tests restructured into TestGeminiJson class.
- #4 tempfile/os.unlink → pytest tmp_path everywhere.
- #5 elif not text → else (the elif branch was dead).
- #6 Module docstring updated to mention Google AI Studio.
Tests: 9 new cases in TestGeminiJson covering all three layouts,
multi-part text joining, non-text part skipping, has_model_role
disambiguation, dispatch-chain regression for review #1.
* feat(normalize): add Continue.dev session parser
Add _try_continue_json() normalizer for Continue.dev AI assistant
sessions (~/.continue/sessions/*.json). Parses history array with
role/content pairs, handles tool calls, system messages, and metadata.
Closes #59 (partial — adds Continue.dev format support)
Includes comprehensive test coverage for valid sessions, edge cases,
malformed input, and unicode content.
* fix: preserve collection name on MCP search retry
* feat: add Cursor IDE support (hooks, plugin, skill, docs, tests)
Adds first-class Cursor IDE integration alongside the existing Claude
Code and Codex hook flows, so Cursor users get the same automatic
diary saves, pre-compaction transcript capture, and session-start
memory recall — without changing any default behaviour for existing
users.
What's included
---------------
Cursor hook scripts (hooks/cursor/):
- mempal_save_hook_cursor.sh — Stop event, counter +
loop_count guard, pending-save marker consumption, background
mempalace mine, followup_message emission.
- mempal_precompact_hook_cursor.sh — synchronous mine before
compaction, drops a pending_save marker, returns user_message.
- mempal_wake_hook_cursor.sh — sessionStart event,
wing-scoped recall guidance via additional_context.
- lib/common.sh — shared parsing + state helpers
(bash 3.2 safe, no heredoc-in-subshell traps).
- install.sh — idempotent installer with
--scope, --variant, --dry-run, --uninstall. Recognises existing
entries by basename so re-installs across paths work.
- STDIN_SHAPE.md, README.md — payload schemas + quick
reference.
Cursor plugin (.cursor-plugin/ + repo-root components):
- plugin.json, marketplace.json, README.md.
- skills/mempalace/SKILL.md — model-invocable skill mirroring the
Claude plugin's skill surface.
- commands/mempalace-{help,init,mine,search,status}.md — slash
commands for marketplace-published installs (filename = slug).
- mcp.json — auto-registers the mempalace MCP
server, wrapped under the documented mcpServers key.
Examples + docs:
- examples/cursor/hooks.json, hooks.minimal.json + README.
- website/guide/cursor-hooks.md + sidebar entry.
- README.md and CHANGELOG.md updates.
Tests (129 new, all green):
- tests/test_cursor_hooks_shell.py — 75 behavioural tests for
the three hook scripts: kill switches, input parsing, counter
logic, loop prevention, pending markers, wing inference, logging.
- tests/test_cursor_hooks_install.py — 19 contract tests for the
installer: dry-run, idempotent merge, basename-matched uninstall,
refusal to overwrite malformed JSON.
- tests/test_cursor_plugin_manifest.py — 35 contract tests for the
plugin: manifest validity, version sync with mempalace.version,
mcp.json shape, skill/command frontmatter, default-discovery
layout invariants.
Design notes
------------
- Local-first and zero-API by default; hooks never call external
services. Same privacy model as the existing Claude Code hooks.
- Fail-open: hook scripts deliberately do not use set -e so a broken
hook can never block the user's conversation.
- Cursor preCompact cannot block + return a followup, so we
synchronously mine the transcript and drop a pending_save marker
that the next stop hook consumes — guarantees verbatim capture
before context window compression.
- Cursor's default plugin discovery requires real commands/, skills/,
and mcp.json at the plugin root (verified against the cached
cloudflare plugin); .cursor-plugin/{commands,skills} are convenience
symlinks back to those canonical locations.
- bash 3.2 compatibility throughout: avoids heredoc-in-command-
substitution parser bugs; uses python -c for JSON parsing;
basename-matched entry recognition in install.sh.
- All changes are additive. No existing files are removed, no
existing hooks change behaviour, and no new runtime dependencies
are introduced.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cursor): address gemini-code-assist review on PR #1632
Five fixes from the Gemini Code Assist review on
https://github.com/MemPalace/mempalace/pull/1632 — three real bugs,
two cleanups, all consistent with the bash-3.2-compatibility
contract documented in the original commit.
Bug fixes (high)
----------------
1. hooks/cursor/lib/common.sh — config.json kill-switch check used
a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)`
command substitution. The heredoc body contains parens which
trips the macOS bash 3.2.57 parser bug. Replaced with a
`python -c '...'` call passing the config path as argv[1]. Matches
the pattern already used in mempal_parse_stdin in the same file.
2. hooks/cursor/install.sh — a relative `--install-dir` was written
verbatim into hooks.json. Cursor invokes hook commands from its
own working directory (typically the project root), so a relative
command path would silently fail to launch the hook. Now resolved
to an absolute path against `$PWD` before being baked in.
3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0`
would crash bash on `$((NEXT % 0))` (division by zero). Extended
the existing sanitiser case to coerce 0 to the default interval
alongside empty / non-numeric values.
Cleanups (medium)
-----------------
4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now
inlined as `python -c '...'`. Removes a small leak window
(tmpfile would linger if the script were interrupted between
mktemp and rm -f) and shortens the script.
5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different
semantics on BSD (macOS) vs GNU mktemp. Switched to the
portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"`
which behaves identically on both.
Regression tests
----------------
- tests/test_cursor_hooks_shell.py
test_save_interval_zero_is_coerced_to_default — guards fix #3.
- tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath
class:
test_relative_install_dir_is_absolutized_in_hooks_json — guards
fix #2 against regression.
test_absolute_install_dir_is_preserved_verbatim — guards that
the relative-to-absolute resolution does not mangle paths
that were already absolute.
Verification
------------
- bash -n on all three edited scripts: clean.
- uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests).
- uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed,
3 skipped (pre-existing).
- uv run ruff check . / ruff format --check .: clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cursor): address igorls review on PR #1632
Resolves the maintainer review on the Cursor IDE support PR. Cursor-only
scope; cross-IDE items (wing-naming convention, shared-file merge order)
are coordinated on the separate Antigravity branch.
followup_message default (the one "decide before merge" item):
- Keep the stop-hook followup ON by default. Cursor's transcript format
is undocumented and mempalace/normalize.py has no Cursor parser, so the
background `mempalace mine --mode convos` is best-effort only and does
not yet yield clean verbatim drawers. The followup is therefore the
load-bearing verbatim-capture path; defaulting it off would leave a
default Cursor install capturing nothing.
- Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for
users who want the Claude-style "zero tokens in chat" behaviour. The
hook still mines and keeps its counters/markers when silenced.
- Correct the misleading "background mine captures it" comments in the
save and precompact hooks; update hooks/cursor/README.md and the guide.
Hygiene fixes:
- Drop the hardcoded "version" field from .cursor-plugin/plugin.json and
marketplace.json (mempalace/version.py is the single source of truth);
tests now assert the field stays absent.
- Remove the committed .cursor-plugin/{commands,skills} symlinks (they
break on Windows clones with core.symlinks=false and were redundant
with the real repo-root components that `source: "."` already serves);
add a guard test that no symlinks exist under .cursor-plugin/.
- Document the preCompact synchronous-mine timeout tradeoff and that an
incremental/append-only mine is recoverable if killed (no corruption).
- Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS,
default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only,
after the kill-switch check; shared logs and antigravity_* are untouched.
Verification: full suite green (2424 passed, 3 skipped), ruff check +
format clean, bash -n clean on all cursor scripts. +30 Cursor tests
(followup opt-out, state GC, TTL validation, no-symlink/version guards).
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(tests): apply ruff 0.4.x format to test_normalize
Fixes lint CI: ruff format --check flagged blank-line and long-dict
wrapping in the Continue.dev parser tests.
* fix(searcher): scope neighbor expansion by parent_drawer_id (#1580)
* fix(mcp): drop top-level anyOf from diary_write schema
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.
The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.
Closes #1711
* docs(openclaw): catch up SKILL.md with 8 newer MCP tools
The openclaw skill was last updated when mempalace exposed 19 MCP
tools. Since then 13 more agent-facing tools have landed; this PR
documents the 8 that openclaw should expose so agents can call them
natively instead of falling back to `npx mcporter call ...`:
Search & Browse:
- mempalace_list_drawers (paginated drawer listing)
- mempalace_get_drawer (fetch a single drawer by id)
Palace Graph:
- mempalace_create_tunnel (explicit cross-wing link)
- mempalace_list_tunnels (enumerate explicit tunnels)
- mempalace_delete_tunnel (remove an explicit tunnel)
- mempalace_follow_tunnels (walk explicit tunnels from a room)
Write / Session:
- mempalace_update_drawer (mutate content or relocate a drawer)
- mempalace_memories_filed_away (ack the silent auto-save hook)
The 3 admin-only tools (mempalace_sync, mempalace_hook_settings,
mempalace_reconnect) are intentionally left out — they're host/admin
operations, not agent-facing memory operations. The Hermes
MemoryProvider plugin landing in MemPalace/mempalace#1684 makes the
same call.
Version bumped 3.3.0 -> 3.4.0 (additive tool surface, no breaking
changes to existing tool docs).
* docs(openclaw): address review round 1
- Fix mempalace_find_tunnels params: (required) -> optional. The MCP
handler defaults both wing_a and wing_b to None
(mempalace/mcp_server.py:1277), so the prior docs were factually
wrong. Caught by gemini-code-assist on PR #1719.
- Clarify implicit-vs-explicit tunnel distinction with consistent
casing and a brief in-line definition (implicit = discovered from
drawer content overlap; explicit = user/agent-declared link).
Suggested by copilot-pull-request-reviewer.
- Split the mempalace_memories_filed_away one-liner into a short
description plus 'Returns' and 'When to call' sub-bullets for
readability. Suggested by copilot-pull-request-reviewer.
* fix: detect Java project manifests
* fix: handle rootless Java subprojects
* fix(mcp): fail closed when add_drawer idempotency pre-check fails
* feat: add mempalace-recall skill and optional Cursor recall rule
Ports the OpenClaw "search before answering" protocol to the Cursor and
Claude plugin surfaces so the agent reads the palace before answering
about past work, people, projects, or prior decisions instead of
guessing from model memory.
- integrations/shared/recall-protocol.md: single source of truth for the
recall protocol, referenced by the skill and the rule so they cannot
drift.
- skills/mempalace-recall/SKILL.md: recall-only skill (the mempalace
skill keeps setup/mine/status); cross-linked from the ops skill.
- rules/mempalace-recall.mdc: plugin recall rule, alwaysApply: false so
it only fires on recall-relevant turns and never adds MCP latency to
greenfield work.
- examples/cursor/rules/: opt-in copies for non-plugin users, including
an aggressive alwaysApply: true variant documented with its latency
tradeoff.
- .claude-plugin/skills/mempalace-recall/SKILL.md: Claude plugin parity.
- tests: assert the recall skill and rules/ discovery layout; the
shipped rule must be alwaysApply: false.
- docs: .cursor-plugin/README.md and the cursor-hooks guide now describe
the three layers of recall (hook + skill + rule).
The Antigravity plugin mirror lands as a follow-up on the antigravity
branch, where .antigravity-plugin/ exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(repair): run post-rebuild FTS5 cleanup on legacy cmd_repair path (#1747)
A clean `mempalace repair --yes` (legacy path) finished without
_vacuum_and_rebuild_fts5: the bulk delete_collection + re-upsert cycle
leaves the FTS5 inverted index inconsistent, so the next repair aborts
at the sqlite integrity preflight. rebuild_index() got this cleanup
when #1517 was fixed; cmd_repair never did.
Extract the shared epilogue _post_rebuild_cleanup() (close chroma
handles, then VACUUM + rebuild FTS5) and call it from both full-rebuild
paths so they cannot drift apart again. Cleanup runs on the legacy
success path only; failure/restore paths are unchanged.
Closes #1747
Co-Authored-By: nord- <3777600+nord-@users.noreply.github.com>
* test(backends): live-substrate conformance module for pgvector
Mirrors the portable fake-client arms of test_pgvector_backend.py
against a real PostgreSQL+pgvector server and adds live-only arms the
in-memory fake cannot exercise: real <=> operator ground truth, JSONB
pushdown vs local-fallback equivalence, cross-namespace isolation on
real tables, 8-connection concurrent writers, and the advisory-lock
serialization of run_maintenance('reindex') under a 2-connection race.
Gated on MEMPALACE_PGVECTOR_LIVE_DSN (same pattern as the qdrant live
gate); skips cleanly when unset. First run: 15/15 pass on PostgreSQL
16.10 + pgvector 0.8.2 (+AGE 1.6.0 in the same server), psycopg 3.3.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review(gemini): marker-race stub, created-list lock, exact-order + exactly-one-ran asserts
- Stub _write_marker on the 8 concurrent writer backends: upsert()
rewrites the marker on every call with a plain open('w'), so backends
sharing one local_path race on the same file (sharing violations on
Windows) — a test-design artifact, not the contract under test
- Guard the fixture's created list with a lock for the threaded tests
- Assert exact distance-ordered ids in the query/filter arms
- Reindex race: exactly one 'ran' (index absent beforehand, so the
advisory-lock winner must build)
Re-run live after changes: 15/15 pass (PG 16.10, pgvector 0.8.2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(embedding): chunk EmbeddinggemmaONNX batches to bound ONNX memory (#1770)
One session.run over a repair-scale batch (5000 docs) allocates
attention buffers far beyond available RAM and the kernel kills the
process. Mirror chromadb's ONNXMiniLM_L6_V2 and embed in sub-batches
of 32; per-chunk padding also stops one long doc inflating the whole
batch.
Co-Authored-By: mojie5 <262519016+mojie5@users.noreply.github.com>
* fix(embedding): lock EF cache and lazy load, guard inputs (#1770)
Two threads sharing a cold EmbeddinggemmaONNX via _EF_CACHE could each
build a full model session, and two factory callers could each keep a
private instance. The load is now double-check locked with the session
published last, and the factory cache has an atomic check-then-construct
behind a lock-free fast path. __call__ wraps a bare string, returns []
for None and empty input before the lazy download, and its annotation
matches the accepted types.
* fix(hallways): scope hallway-file path to MempalaceConfig.palace_path (#1778)
Pre-3.4 the hallway store was hardcoded at ~/.mempalace/hallways.json
regardless of the configured palace_path, so two palaces on one host
silently shared one file. Mining into palace-A leaked records into
palace-B's hallway code paths.
Apply the 3.3.6 tunnel-file migration pattern:
* MempalaceConfig.hallway_file resolves to <dirname(palace_path)>/hallways.json
* hallways._get_hallway_file(config) reads through MempalaceConfig
* hallways._legacy_hallway_file() exposes the pre-migration hardcoded path
for one-time orphan detection; _load_hallways logs a one-line warning
when the legacy file exists but the configured one doesn't, matching
palace_graph._load_tunnels behavior. No auto-migration — silent merging
risks clobbering newer data.
Atomic-write + 0600 semantics unchanged. Module-level _HALLWAY_FILE
constant kept and honored when monkey-patched directly, so the three
existing test sites that patch it (test_hallways.py, test_hallways_pagination.py,
test_mcp_server.py) keep working without modification.
New coverage in tests/test_hallways_palace_scoped.py mirrors the analogous
tunnel tests: resolver default + custom palace_path + env-var redirect,
orphaned-legacy warning + no-warning when paths match, and an end-to-end
multi-palace isolation regression guard.
Closes #1778
* fixup(hallways): drop _HALLWAY_FILE back-compat shim, migrate existing tests to resolver
Replaces the back-compat shim in _load_hallways/_save_hallways (which
honored direct monkey-patches of the _HALLWAY_FILE module constant) with
a clean single-source-of-truth resolver, matching the palace_graph
tunnel-file migration in 3.3.6.
The three existing test sites (tests/test_hallways.py,
tests/test_hallways_pagination.py, tests/test_mcp_server.py) now
monkey-patch _get_hallway_file and _legacy_hallway_file directly,
exactly mirroring the helper in tests/test_palace_graph_tunnels.py.
Production code now has one branch through the path resolution instead
of two. No behavior change. 269/269 hallway + tunnel + mcp-server
tests pass on Python 3.11 and 3.12, ruff clean.
* fixup(hallways): address gemini-code-assist review on PR #1780
Two catches on tests/test_hallways_palace_scoped.py
TestMultiPalaceIsolation.test_save_then_load_under_different_palace_returns_empty:
1. Stale comment referencing the removed _HALLWAY_FILE back-compat shim
(deleted in the prior fixup commit). Removed.
2. _legacy_hallway_file was not monkey-patched, so the test isolation gap
let _load_hallways check the host's real ~/.mempalace/hallways.json
when evaluating the legacy-warning branch. Now patched to a tmp_path
sibling, matching the helper pattern used in test_palace_graph_tunnels.
* fix(ids): use length-prefixed recipe v3
* fix(mcp): treat chunked drawers as logical drawers
* fix(mcp): avoid mutating drawer metadata
* fix(ids): simplify v3 length-prefixed hashing
* chore(deps): bump docker/metadata-action from 5 to 6
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)
---
updated-dependencies:
- dependency-name: docker/metadata-action
dependency-version: '6'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump docker/build-push-action from 6 to 7
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)
---
updated-dependencies:
- dependency-name: docker/build-push-action
dependency-version: '7'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)
---
updated-dependencies:
- dependency-name: docker/login-action
dependency-version: '4'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(hooks): normalize Windows transcript paths in shell hooks
* fix(backends): serialize first connect in sqlite_exact and pgvector (#1774, #1775)
Co-Authored-By: jphein <19301265+jphein@users.noreply.github.com>
* fix(hooks): preserve fail-loud parse diagnostics
* fix(hooks): typo regression addressed
* fix(tests): run fact_checker __main__ via subprocess to clear runpy warning
`tests/test_fact_checker.py` imports symbols from `mempalace.fact_checker` at
module top (putting it in sys.modules), then `TestCLI.test_exits_nonzero_when_
issues_found` re-executed the same module as __main__ via
`runpy.run_module("mempalace.fact_checker", run_name="__main__")`. runpy warns
because it re-runs an already-imported module against a half-initialized state:
RuntimeWarning: 'mempalace.fact_checker' found in sys.modules after import of
package 'mempalace', but prior to execution of 'mempalace.fact_checker'
Run the CLI in a fresh process via `subprocess.run([sys.executable, "-m",
"mempalace.fact_checker", ...])` instead — no sys.modules collision, and it
exercises the real `python -m` entry point. Assertions are preserved
(SystemExit code 1 → returncode 1; captured stdout substring → result.stdout).
The child's entity registry (`~/.mempalace/known_entities.json`, resolved via
expanduser at import) is redirected by overriding both HOME and USERPROFILE in
the subprocess env so it works on POSIX and Windows.
Verified: `pytest tests/test_fact_checker.py -W error::RuntimeWarning` passes
(26) with the warning promoted to error — proving it no longer fires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): avoid Chroma open when cached DB disappears
* test: stabilize release validation on develop
* fix(palace): clean source mine locks safely
* fix: close blob seq sqlite migration connection
* fix: address mine lock review feedback
* test: stabilize closet boost fixture on Windows
* chore(release): 3.4.1
Bump version across all sources (version.py, pyproject.toml, both
Claude plugin manifests, Codex plugin manifest, README badge, uv.lock)
and promote the Unreleased changelog to 3.4.1.
Shipping: Cursor IDE plugin + hooks, first-class Antigravity IDE
support (with zero-config interpreter resolution), embeddinggemma
bulk re-embed OOM fix, and backup-retention pruning.
Also rebuilds the CHANGELOG compare-link block, which had been left
at v3.2.0: adds the full 3.3.0-3.4.1 chain plus the previously
undocumented 3.4.0, and points Unreleased at v3.4.1...HEAD. Every
version header now resolves to a compare link.
* fix(hooks): portable mtime in macOS hook throttles; doc cleanup
Address review feedback surfaced on the 3.4.1 release promotion (#1810).
Bug fix — `date -r FILE` is GNU-only. On BSD/macOS `date -r` expects
epoch seconds, not a path, so the staleness/throttle checks in the new
Cursor and Antigravity hooks silently failed on macOS: the state GC
swept on every fire and the pending-save guard was skipped. Replace
with a portable `os.path.getmtime` one-liner via the already-resolved
$MEMPAL_PYTHON_BIN (cursor/lib, antigravity/lib, antigravity save hook).
This restores the "bash 3.2.57 / macOS default" compatibility the
Antigravity changelog claims.
Docs:
- Correct the MCP tool count to 33 (was 19/29/31 in 21 places across
plugin manifests, READMEs, and website docs — all drifted from the
TOOLS dict / mcp-tools.md reference, which both have 33).
- Fix broken CHANGELOG link to the Cursor skill (skills/, not
.cursor-plugin/skills/).
- Fix one-too-many `../` in skills/mempalace/SKILL.md's cursor-hooks
link (resolved above the repo root).
- Add the required `mcpServers` wrapper to the mcp.json example in
.cursor-plugin/README.md so copy-paste yields a valid Cursor config.
Left intentionally unchanged: the os.dup2 fd-1 redirect in
mcp_server.py is deliberate (#225 keeps JSON-RPC off fd 1).
* style(hooks): single-quote the static python -c mtime snippet
The snippet has no shell interpolation — the path arrives via argv, not
string interpolation — so single quotes are correct and make it
unambiguous that nothing is shell-expanded. Behavior is identical:
`sys.argv[1]` contains no `$`, so it was never expanded (verified
empirically). Matches the single-quoted `python -c` blocks already in
hooks/cursor/lib/common.sh. No functional change.
* test(migrate): cover swap-failure rollback
Adds end-to-end regression coverage for the migration swap path where os.replace hits EXDEV, the shutil.move fallback fails, and the original palace must be restored from the rename-aside copy.
* feat(mcp): add mempalace_delete_by_source bulk-cleanup tool (#1722)
Adds an MCP tool to remove every drawer mined from a given source_file
exact match, for cleaning up benchmark/test data accidentally mined into
a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config
JSON) that drowns out real memories in semantic search.
Matching is pushed to the backend via delete(where={"source_file": ...})
the same idiom the miner and diary-ingest paths already use so it is not
subject to the SQLite variable limit regardless of how many drawers share
the source. Defaults to a dry run reporting match count and a sample;
dry_run=false commits. Absent source is an idempotent no-op, not an error.
* fix(mcp): harden delete_by_source per review — strip surrogates + type guard
Address Gemini review on #1729:
- normalize source_file with strip_lone_surrogates so exact matching hits
rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring
tool_add_drawer's ingestion-side normalization
- isinstance(str) guard so a non-string source_file returns a clean error
instead of AttributeError
- default missing wing/room to "" in the dry-run sample, consistent with
the rest of the file
- add tests: non-string rejection + surrogate-normalization match
* feat(miner): add PHP ecosystem file extensions
* fix(claude-plugin): run final mine on SessionEnd
* fix reviewer feedback: To prevent a KeyError and provide a clear, actionable assertion failure message if PreCompact is ever missing
* fix(chroma): route stale hnsw divergence to sqlite fallback
* fix reviewer feedback for chroma and tests
* fix(mcp): refuse second writer for same palace
* fix(mcp): cache writer lock setup failures
* feat: add opt-in local daemon for queued MemPalace writes
- New mempalace/daemon.py: long-lived localhost HTTP server (127.0.0.1) with a
SQLite WAL job queue, single worker thread, bearer-token auth, and owner-only
file perms (0600/0700) on queue DB, token, endpoint, and log.
- New mempalace/service.py: transport-neutral job execution surface shared by the
daemon, with per-job env isolation so one job's backend/palace switch cannot
leak into the next. mcp_tool is allowlisted to write-classified tools only.
- Crash recovery re-queues jobs left 'running' by a killed daemon; jobs that
already exhausted MAX_ATTEMPTS are dead-lettered to 'failed' instead of being
retried (non-idempotent diary_write would otherwise duplicate verbatim
content on every restart).
- Bounded retention prunes terminal jobs older than 7 days
(MEMPALACE_DAEMON_RETENTION_DAYS); queued/running jobs are never touched so a
crash mid-prune cannot drop in-flight work.
- CLI: --daemon/--background on mine/sync submit to the queue; new
`mempalace daemon {start,stop,status,jobs,wait}` subcommand. Strictly opt-in:
no flag, env, or config means no daemon and no behavior change.
- Hooks opt in via MEMPALACE_HOOKS_DAEMON or config hooks.daemon; when the daemon
is not already running, hooks fall back to the existing direct/spawn path so
the 500ms hook budget is preserved (hooks never auto-start the daemon).
- service.run_sync renders the same operator-facing report shape as the direct
CLI sync path (no_source, out_of_scope, by_source, Re-run/Removed hints) and
drops the old KeyError-prone 'deleted' read.
* fix(mcp): gate startup on sqlite integrity failures
* chore(deps-dev): bump ruff from 0.15.15 to 0.15.18
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.15 to 0.15.18.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.15...0.15.18)
---
updated-dependencies:
- dependency-name: ruff
dependency-version: 0.15.18
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(mcp): applied 3 of the 4 reviewer suggestions
* fix(mcp): guard remaining None palace_path in _mcp_sqlite_integrity_refusal. Added one regression test calling the function directly with palace_path=None.
* fix: unblock daemon PR CI + address review comments
CI was red on all three platforms for the daemon-mode draft PR. Root causes
and fixes:
- Linux 3.9 collection error: `_submit_daemon_job`'s `dedupe_key: str | None`
parameter annotation is evaluated at def time, and hooks_cli.py has no
`from __future__ import annotations` — `str | None` raises TypeError on 3.9.
Reverted to `dedupe_key: str = None` (the original, 3.9-safe). The other
`int | None` in the file is a function-local annotation, which is never
evaluated, so it was never the problem.
- macOS/Windows daemon lifecycle flakes: the 3 HTTP-lifecycle tests failed at
the 10s readiness deadline on contended CI runners (localhost bind is
sub-second locally but took ~5s when it passed on the macOS fleet, >10s when
it didn't), and because the server thread never shuts down on timeout,
run_server's `os.environ["MEMPALACE_PALACE_PATH"]` + `os.umask(0o077)`
mutations leaked into the rest of the suite — poisoning every later test that
reads MempalaceConfig().palace_path (the 60+ test_mcp_server cascade on macOS;
the at-exit socket hang → SIGINT on Windows). Bumped the readiness deadline
to 30s and added a module-scoped snapshot + autouse fixture in test_daemon.py
that force-restores the env + umask to the pre-suite baseline after every
daemon test, so a leaked server thread can't poison other test files.
Gemini review comments (fixed in code, no thread replies per convention):
- daemon.py `_connect()` was a bare `sqlite3.connect` whose `with`-block only
managed the transaction, not the connection — an unbounded FD leak in a
long-lived daemon running thousands of jobs (also the source of the Windows
"unclosed database" ResourceWarning noise). Converted to a closing
@contextlib.contextmanager.
- `QueueStore.finish()` gained `only_if_running`; `_safe_finish` passes it so a
late worker finish can't overwrite a shutdown-cancelled job back to
succeeded/failed — removes the reliance on process-exit timing.
- `DaemonClient.request` wraps the final `json.loads` in try/except
JSONDecodeError → DaemonError, so a non-JSON 2xx response surfaces as a
structured error instead of a bare JSONDecodeError.
- test_sync.py: removed the module-level `import mempalace.mcp_server` and moved
the stdout-rebinding side effect into an autouse fixture scoped to
TestServiceRunSyncReport, so the embedder/Chroma import chain is no longer
forced at collection time for the existing sync tests.
Coverage: added focused happy-path tests for service.run_sync early-returns,
run_mine backend application + invalid mode, execute_job kind dispatch,
run_diary_write arg forwarding, run_mcp_tool write-tool dispatch, and
print_job_result — lifts service.py from 57% to 85% so the new files
(service 85%, daemon 80%) don't drag the total below the 80% CI gate now that
the daemon tests complete and the gate is actually evaluated.
* fix: daemon client bypasses proxy discovery; tests force-shutdown server thread
DaemonClient.request now uses a no-proxy opener (build_opener(ProxyHandler({})))
instead of urllib.urlopen. The daemon is always on 127.0.0.1, so a request must
never go through an HTTP proxy — this is the correct production choice. It also
bypasses urllib's proxy discovery (macOS _scproxy via SystemConfiguration), which
runs on the first request to any host and is NOT bounded by the per-request
timeout: on a CI runner with no network it hangs for tens of seconds, which looked
exactly like the daemon never came up (test_daemon_http_lifecycle_executes_job
timed out at 30.18s). With the no-proxy opener the lifecycle runs in 0.78s and no
server thread is leaked — which also removes the timing skew that made the
sqlite_exact concurrent-connection test flake on macOS CI.
The leaked server thread was also the Windows exit-hang root cause: a slow/failed
client.shutdown() POST left serve_forever running, and the interpreter blocked on
the open listening socket at process exit. Tests now capture the httpd run_server
creates (by subclassing daemon.ThreadingHTTPServer) and force httpd.shutdown() +
server_close() from the test thread if the normal shutdown path leaves the thread
alive, asserting the thread died so a leak becomes a visible failure instead of
a silent exit hang.
* test: win32-only diagnostic for daemon process-exit hang
The Windows CI run passes all 666 tests then hangs at interpreter shutdown
(KeyboardInterrupt at socket.py:723) until the runner kills it. All daemon
lifecycle tests assert their server threads died, so the hang is a different
non-daemon thread blocked on a socket — not the daemon server thread. CI
round-trips can't show which thread it is.
Add a win32-only session fixture that:
- arms faulthandler.dump_traceback_later(130s) to print every thread's stack
to stderr once the hang has run a while, and
- prints every live thread (name + daemon flag) at session teardown — a
non-daemon thread present there is the shutdown blocker.
Gated to sys.platform == 'win32' so Linux/macOS CI see no extra output. Remove
once the Windows hang is fixed.
* fix(daemon): Windows-safe pid liveness probe; finalize cross-platform daemon tests
_pid_alive used os.kill(pid, 0) as an existence check. On Windows signal 0
is signal.CTRL_C_EVENT, so Python routes it to GenerateConsoleCtrlEvent and
sends a console Ctrl-C to the target's process group rather than probing the
pid. DaemonClient polls a same-process endpoint during startup, so on a CI
runner with an attached console that Ctrl-C was delivered back to the
interpreter as a spurious KeyboardInterrupt — the Windows CI hang that
interrupted the suite at the first daemon HTTP-lifecycle test (socket.py
recv). Probe via the Win32 OpenProcess/WaitForSingleObject handle API instead,
which has no signalling side effects. This is also a real Windows production
bug, not just a test artifact.
Tests:
- Skip the two owner-only (0600) permission tests on Windows: os.chmod cannot
represent POSIX mode bits there (files report 0o666); the daemon relies on
user-profile ACLs on Windows.
- _start_server now captures and re-surfaces a run_server thread crash instead
of spinning for 30s and failing with a bare assert (diagnoses the macOS
startup flake).
- Add a regression test asserting _pid_alive is correct and emits no console
control event when hammered like the poll loop.
- Remove the temporary win32 exit-hang diagnostic fixture from conftest now
that the root cause is fixed.
* fix(daemon): skip reverse-DNS in server_bind so startup can't block ~30s
HTTPServer.server_bind() resolves server_name via socket.getfqdn(host). For the
daemon's 127.0.0.1 bind that lookup is pointless, and on a host with slow or
absent reverse DNS it blocks startup until the resolver times out (~30s) — which
looks exactly like the daemon never coming up. This is why the first daemon
HTTP-lifecycle test timed out on the macOS CI runner (httpd_bound=False after
30s) while every later one bound in seconds once the OS had cached the negative
lookup. Bind via TCPServer directly and set server_name from the literal host.
* feat(hooks): add a budget-safe SessionEnd save hook for clean exits (#1341)
Short sessions that exit cleanly below SAVE_INTERVAL and without a PreCompact
were never saved. Add a SessionEnd hook that takes one final flush.
Claude Code budgets SessionEnd hooks at 1.5s and a plugin-provided timeout
cannot raise it, and a cold mempalace start exceeds that, so the wrapper
backgrounds the work and returns immediately; the detached child completes the
transcript ingest, project mine, and diary checkpoint after the session exits.
The handler validates transcript_path through _validate_transcript_path before
any ingest or diary write, so a traversal or wrong-suffix path is rejected while
the independent project mine still runs.
Adds hook_session_end, both shell wrappers, the plugin hooks.json entry, the
session-end CLI choice, and focused tests.
(cherry picked from commit 10e1450e04fc7cec72984ab7442d3b4fca1490e8)
* fix(claude-plugin): resolve SessionEnd merge semantics
* fix(claude-plugin):reviewer feedback for _validate_transcript_path function calls Path.resolve(), which can raise an OSError
* fix(daemon): address post-merge review feedback on #1826
Five fixes from the Copilot review of the merged daemon PR:
1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed
verbatim payloads but were created with the caller's umask. Set the
owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore
(not only once the HTTP server starts), and harden any existing sidecars in
QueueStore._init_db as defense-in-depth.
2. DoS guard: reject a negative Content-Length in the request reader.
rfile.read(-1) would block until the client disconnects and bypass the
MAX_BODY_BYTES cap.
3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS)
into a new side-effect-free mempalace/wal.py. The CLI sync path and the
daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`,
which runs mcp_server's import-time stdio protection (os.dup2(2, 1);
sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output.
mcp_server/cli/service now import from mempalace.wal.
4. Correctness: run_mcp_tool treated any dict as success. Write tools that
return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel
validation) were recorded as succeeded; now the "error" key infers failure.
5. Hook budget: get_client_if_running()/health() take an explicit timeout, and
the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s)
so a wedged daemon can't stall the hook for the default 5s.
Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the
umask ordering, negative Content-Length, run_mcp_tool error inference, and the
short probe timeout.
* fix(backends): single-scroll bulk metadata fetch for Qdrant; bump scroll page size (#1796)
* fix: apply suggested reviewer suggestions
* updated tests/test_qdrant_bulk_metadata_scroll.py because of CI failure after push the 2nd commit
* perf(embedding): cap ORT intra-op threads so a background mine doesn't pin every core (#1068)
ChromaDB's ONNX embedder builds its InferenceSession without a thread cap, so
ORT's intra-op pool defaults to the physical core count. OMP_NUM_THREADS is
inert against it (ORT owns its own pool), so a background `mempalace mine`
pins 4-5 cores and stacked Stop-hook fires turn the machine into a thermal
event.
Add an `embedding_threads` config knob (env MEMPALACE_EMBEDDING_THREADS or
config.json). Unset/"auto" caps the intra-op pool at half the logical CPUs so
a fresh install stays usable out of the box; a positive integer sets an exact
count; 0/negative leaves ORT uncapped for users who want max throughput.
The cap is applied via SessionOptions at session construction:
- `_MempalaceONNX` (default minilm) overrides the `model` cached_property to
rebuild the session the same way upstream does plus the cap, falling back to
upstream's uncapped build if chromadb internals shift.
- `EmbeddinggemmaONNX` builds its session through the shared
`_intra_op_session_options()` helper.
* perf(mcp): answer overview tools from the sqlite aggregate to fix large-palace timeouts (#1748, #1379)
tool_status / list_wings / list_rooms / get_taxonomy paged the entire
collection metadata through the chroma client (`_fetch_all_metadata`, a
1000-row offset loop), which cold-loads the HNSW index and materializes
hundreds of MB of dicts. On six-figure palaces these exceed the MCP host
tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s
metadata cache only dedups repeat calls — it does not stop the cold-call
timeout.
A correct single-query SQL cross-tab already exists
(`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default
(`miner.status`), but the MCP tools never used it — and the MCP-side sqlite
reader only ran behind the `vector_disabled` recovery path.
Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to
fall back) and wire it as the default path into all four overview tools. They
now answer from one GROUP BY without touching HNSW. Non-chroma backends
(qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the
existing client path unchanged.
graph_stats (also named in #1379) builds an in-memory graph via build_graph()
and needs its own treatment — tracked separately.
* fix(pgvector): strip NUL bytes so a transcript NUL no longer aborts the mine (#1829)
PostgreSQL cannot store NUL (0x00) in text or jsonb. On the pgvector write path
a NUL in `document` is rejected by psycopg ("PostgreSQL text fields cannot
contain NUL (0x00) bytes") and a NUL in `metadata` becomes a JSON unicode escape
the jsonb cast rejects ("unsupported Unicode escape sequence"). `_execute`
re-wraps either as BackendError and `_mine_impl` re-raises, so the whole mine
exits non-zero and every file after the offending one is left unmined. ChromaDB,
SQLite, and Qdrant store the byte verbatim, so only pgvector hard-fails.
Add a recursive `_strip_nul` helper and apply it to id, document, and metadata
in `_PgVectorClient.upsert_rows`, mirroring the backend-layer sanitization
`_sanitize_documents_for_chromadb` already does for lone surrogates on the same
bulk-ingest paths. ids are SHA-256 hashes and metadata keys are fixed field
names, so the id and key passes are no-ops in practice; only transcript-derived
values change.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
* fix: address PR review feedback (preserve "unknown" label; use super().model)
#1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None)
back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room
metadata keep the client path's output contract — no observable API change
for MCP clients on legacy/partial drawers.
#1068: invoke the parent embedder build via super().model instead of reaching
into cached_property's .func attribute, so the uncapped/fallback path survives
chromadb changing `model` to a plain @property or other descriptor.
* perf(mcp): sqlite fast path for graph_stats to fix large-palace timeouts (#1379)
tool_graph_stats built the whole palace graph via build_graph(), which pages
every metadata row (col.get limit/offset) and cold-loads the HNSW index — the
remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings /
list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds
an in-memory graph rather than a flat tally).
Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3,
reconstructing build_graph's room_data and the same stats (total_rooms,
tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same
per-drawer filter (room present, != "general", wing present) and edge
semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend()
guard + client-path fallback as the #1748 overview tools.
Test seeds a real chroma palace mirroring the build_graph parity case in
test_palace_graph, with a tripwire on graph_stats proving the fast path runs
and that "general"/wing-less drawers are excluded. Idea adapted from #1381's
_sqlite_graph_stats.
* fix: address PR review feedback on graph_stats sqlite fast path (#1379)
- Soft-fallback on any exception, not just sqlite3.Error, so an unexpected
schema shape tripping the reconstruction degrades to build_graph() instead
of raising — matching the sibling sqlite fast paths (Copilot).
- Guard an empty/None _config.palace_path before building db_path (Gemini).
- Test: tripwire _get_collection in addition to graph_stats, directly
asserting the fast path never opens the chroma client / cold-loads HNSW
(Copilot).
* fix: percent-encode sqlite read-only URIs so spaced/special-char paths open
sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) mis-parses paths
containing spaces or other URI-reserved characters — common in real home
directories (a Windows "First Last" user folder, many macOS paths), and made
worse by Windows backslashes. The database silently fails to open and the
read-only fast paths fall back (or error) on those machines.
Add config.sqlite_read_uri(), which percent-encodes the path via
urllib.request.pathname2url (lazy-imported to keep config import light), and
route every read-only sqlite reader through it:
- mcp_server._tool_status_via_sqlite
- searcher BM25 sqlite fallback
- repair (status / scan / max-seq read paths)
- backends/chroma (5 readers: counts, wing/room tally, id maps, etc.)
All previously used the same naive f-string construction. Surfaced as a
gemini-code-assist review note on #1837.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(mcp): route _sqlite_graph_stats through sqlite_read_uri
The graph_stats sqlite reader (#1837) and the sqlite_read_uri encoding fix
(#1838) landed in parallel, so _sqlite_graph_stats was the one reader left on
the naive f"file:{db_path}?mode=ro" construction that mis-parses paths with
spaces/special chars. Convert it too — now every read-only sqlite reader
percent-encodes its path. sqlite_read_uri is already imported in mcp_server
from #1838, so this is a call-site-only change.
* fix(pgvector): push get(limit, offset) pagination into SQL (#1830)
PgVectorCollection.get(limit=, offset=) ignored pagination at the SQL
layer: scroll_rows ran SELECT ... WHERE <where> with no LIMIT/OFFSET, so
get() fetched the whole table and sliced in Python. prefetch_mined_set
pages the whole palace on every mine, so mining was O(N^2) in rows
transferred and Python objects built as the palace grows; every other
paginating caller (exporter, migrate, repair, hallways, closet_llm,
miner, palace_graph) paid the same cost.
Push LIMIT/OFFSET into scroll_rows/_scroll with ORDER BY id (the primary
key) for stable offset pagination. get() uses the pushed path only for an
unfiltered page (no ids, no where/where_document, non-negative bounds); a
filtered get keeps the full-scan path because the metadata @> ... pushdown
is broader than the exact _matches_where re-filter for array/object
values, so that re-filter must run before pagination. Full-scroll callers
pass no bound, so their SQL is unchanged.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
* fix(pgvector): replace lone surrogates so a transcript surrogate no longer aborts the mine (#1833)
A lone UTF-16 surrogate (U+D800-U+DFFF) in transcript content has no UTF-8
encoding, so pgvector's bulk upsert_rows makes psycopg raise UnicodeEncodeError
and the whole mine aborts, leaving later files unmined.
Apply config.strip_lone_surrogates (-> U+FFFD) to id, document, and the
serialized metadata JSON in upsert_rows. json.dumps(ensure_ascii=False) leaves a
metadata surrogate raw in the string, so one pass over the serialized JSON covers
it; NUL, by contrast, json-escapes and must be stripped before serialization
(see #1829). Replace rather than drop, matching ChromaDB's document handling.
Verified end to end against live Postgres + pgvector: before, a surrogate in
document or metadata aborts the mine; after, it ingests and round-trips as U+FFFD.
Fixes #1833
* fix(backends): push sqlite_exact get(limit, offset) pagination into SQL
SQLiteExactCollection.get(limit, offset) fetched the whole collection via
_rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced
in Python, so every paginating caller (prefetch_mined_set, status, exporter,
migrate, dedup, sync, ...) re-scanned the entire table per page, making the
sweep O(N^2) in rows materialized.
Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/
where_document and non-negative bounds); filtered, id, and negative pages keep
the full-scan plus Python-slice path so the post-filter still runs first. SQLite
requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY
rowid keeps pages stable.
* ci: re-trigger checks (unrelated Windows closet flake)
* refactor(qdrant): reuse _rows() in get_all_metadata(); fix sys.modules test pollution
Addresses maintainer review on #1832 (the two non-blocking 🟡 items plus
two 🟢 nits)
* test(mcp_server): add missing _fetch_all_metadata delegation/fallback tests.
* feat(search): add an optional source_file filter to mempalace_search (#1815)
Expose source_file alongside wing/room on mempalace_search. build_where_filter
generalizes to 0/1/2+ clauses and the filter threads through the main vector
path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and
the union lexical path so it never silently no-ops. Matching is on the exact
full stored value; results now expose source_path (the full path) for round
tripping, since the displayed source_file is a basename. The MCP schema gains
the source_file property and a path-tolerant sanitizer rejects null bytes,
lone surrogates, and overlong values.
Fixes #1815
Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com>
* fix(mcp): reject non-string source_file with a clean error (#1815)
A JSON number or boolean passed for source_file is not coerced by the
string schema type, so it reached _sanitize_optional_source_file and
raised AttributeError from .strip() rather than a clean validation error.
Add an isinstance guard that raises ValueError, which tool_search returns
as a structured error. Regression test added.
* ci: re-trigger Windows (flaky closet-boost test)
* fix(repair): point index-read failures to repair --mode from-sqlite (#1843)
When the chromadb compactor cannot apply the WAL into the drawers HNSW
segment (InternalError: Failed to apply logs to the hnsw segment writer),
the legacy repair paths fail on their first Collection.count() read and
advise re-mining from source files. The drawer rows are intact in
chroma.sqlite3, so repair --mode from-sqlite rebuilds them; re-mining
silently drops drawers added via the MCP server and diary entries that
have no source file.
Both legacy read-failure sites (cmd_repair and rebuild_index) now emit
shared guidance pointing at the from-sqlite recovery, worded conditionally
so it also covers a live server or mine still holding the palace open.
Co-Authored-By: undeadindustries <9536461+undeadindustries@users.noreply.github.com>
* fix: use CREATE_NO_WINDOW so Windows hook miner spawns don't flash a console (#1783)
Fixes #1783
* fix: point diverged-index recovery at from-sqlite, not re-mine (#1843)
A diverged HNSW index (for example after a failed chromadb compaction)
leaves the drawer rows intact in chroma.sqlite3 but the vector index out
of sync. Re-mining to recover silently drops drawers added through the
MCP server and diary entries, which have no source file.
- repair-status now recommends `mempalace repair --mode from-sqlite
--archive-existing` when DIVERGED, instead of the generic `mempalace
repair`, and explains why re-mining loses data.
- The shared recall protocol and the recall skills (Cursor + Claude
plugin) document the compactor / "Not connected" recovery path:
stop the server, rebuild from SQLite, verify, restart — never repair
in-process from the agent.
Complements #1847 (legacy repair error messages); does not duplicate it.
Does not close #1843 — MCP reconnect resilience and honest add_drawer
write signalling remain open.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: add Windows backup alternative to corrupt-index recovery (#1843)
Gemini review on PR #1849: the optional palace backup step used the
Unix-only `cp -a`, which fails on Windows. MemPalace ships on win32, so
add a PowerShell `Copy-Item` alternative alongside the macOS/Linux form
and note that `--archive-existing` already moves the old palace aside.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add mempalace_checkpoint batch save tool
Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer
Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the
host UI renders one tool-call card and keeps its spinner up for the whole
save. The new tool reuses the existing single-item handlers, so semantic
dedup, idempotency, and verbatim guarantees are unchanged.
- mcp_server.py: add tool_checkpoint + register mempalace_checkpoint
- service.py: classify mempalace_checkpoint as a write tool
- cursor save hook: followup now drives one mempalace_checkpoint call
- docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep
- tests: checkpoint add/dedup/malformed/registry + classify_tool
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: harden tool_checkpoint input validation
Address PR review: guard untrusted MCP client payloads in
mempalace_checkpoint so a single malformed item cannot raise deep in
sanitization and abort the whole batch.
- coerce dedup_threshold to float
- require wing/room/content to be non-empty strings (skip + record error)
- validate the diary object and entry type, recording errors instead of
silently ignoring a malformed diary
On a genuine dedup-check error we still file the drawer rather than skip:
verbatim recall is the priority and add_drawer's idempotency blocks exact
duplicates. Adds tests for the non-string, dedup-error, and malformed-diary
paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: update Cursor followup assertion for checkpoint tool
The save-hook followup now drives a single mempalace_checkpoint call, so
test_threshold_emits_followup_message must assert that tool name instead
of the old add_drawer/check_duplicate/diary_write trio. Fixes the
test-macos / test-linux CI failures on this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): purge matching closets in delete_by_source (#1722)
delete_by_source removed only the drawers, leaving the matching closets
(the AAAK index layer, keyed independently by source_file) behind as stale
pointers at the now-deleted source. Mirror the closet-purge step used by
sync_palace / purge_file_closets: after the drawer delete, best-effort purge
the closets via push-down delete(where=...) so it survives large palaces and
can never abort an already-committed drawer delete.
Dry run now also reports closet_match_count so the caller sees the full blast
radius; commit reports closets_deleted. Adds tests that seed the closet
collection directly (tool_add_drawer doesn't build closets) and assert the
matching closets are purged on commit and counted on dry run.
* feat(mcp): add opt-in HTTP transport
* fix suggestion of reviewer to avoid a critical race condition and other fixes
* test(mcp): keep HTTP transport tests Python 3.9 compatible
* test(mcp): avoid subprocess flakiness in HTTP transport tests
* test(mcp): bypass proxies in HTTP transport loopback tests
* test(mcp): make HTTP transport loopback tests proxy-free
* test(mcp): make HTTP transport tests network-free
* fix(mcp): move _HTTP_REQUEST_LOCK and _HTTP_MAX_REQUEST_BYTES
* fix(mcp): move _HTTP_REQUEST_LOCK
* fix reviewer: handling JSON-RPC
* fix(tests): rewrite test_mcp_http_transport for Python 3.9-3.13 + Windows
* fix(lint): resolve 7 ruff errors in test_mcp_http_transport
* fix(mcp): harden HTTP transport — DNS-rebinding guard, optional token, real tests
The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by
default, but /mcp was unauthenticated with no protection against a malicious
web page reaching a DNS-rebound localhost server, and its tests reached for
Starlette/uvicorn (not project deps) so they were silently skipped in CI —
the production _serve_http handler had zero coverage.
Hardening:
- Pin the Host header to loopback literals + the bound host on a loopback bind
(DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which
is the operator's call and may sit behind a Host-rewriting proxy.
- Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard);
non-browser MCP clients omit Origin and are unaffected.
- Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare);
required on /mcp, never on /healthz so liveness probes work credential-free.
- Warn loudly when bound to a non-loopback host (palace reachable from network).
Testability:
- Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and
drive the real handler over a loopback socket via stdlib http.client.
- Replace the skipped Starlette reimplementation with 12 tests covering dispatch,
initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and
the Host/Origin/token rejections — no third-party deps.
* ci(test-windows): retry the transient ChromaDB HNSW compaction flake
ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.
Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
* chore(release): 3.5.0
Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex
plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools"
prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a
tool). Add the 3.5.0 CHANGELOG entry.
* fix: tighten local guards and file handling
* fix: restore convo miner scan indentation
* fix: green up CI for hardened file handling
- ruff format llm_client.py and miner.py (lint job)
- _copy_file_no_follow: close src fd if the dst open fails (no leak), and
route the rebuild restore through it so backup + restore share one
no-follow/regular-file path
- update repair tests to assert the unified hardened copy instead of the
removed shutil.copy2 calls; backup paths are now timestamped
- update normalize large-file test to stub fstat (size is checked on the
open fd, not via a pre-open os.path.getsize)
* test(wal): cover crash-safety, idempotent setup, and redaction edge paths
The write-ahead log gained its own module in v3.5.0 but sat at 82% coverage;
the uncovered lines were exactly the failure/guard branches that uphold its
contracts: the cache-hit early return, the restricted-FS chmod/mkdir swallow
paths, and the promise that a WAL write failure is logged and never crashes
the calling tool. Add five tests covering those branches plus the non-string
redaction marker, bringing mempalace/wal.py to 100% and locking the
crash-safety guarantees against regression. Test-only; no production change.
* fix: spawn daemon with CREATE_NO_WINDOW to match hook miner (#1783) (#1857)
daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits.
Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console.
Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests).
* fix(cli): add repair rebuild-index alias (#1670)
* Fix/wing slug special chars (#1852)
* fix: sanitize wing slug for project dirs with special characters
Project folders containing characters outside sanitize_name's set
(e.g. a leading '+') leaked into the derived wing name, producing names
like 'wing_+project' that config.sanitize_name rejects, silently
breaking diary auto-save for that project.
Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall
back to 'sessions' when a name reduces to nothing. Route the three
wing-derivation sites through it.
Tests: unit cases for the helper plus a hypothesis property test
asserting wing_<slug> always passes sanitize_name for any inp…
1 parent a8807c3 commit b46f18d
74 files changed
Lines changed: 4030 additions & 452 deletions
File tree
- .claude-plugin/hooks
- .github/workflows
- deploy
- docs
- hooks/cursor
- integrations/openclaw
- mempalace
- backends
- scripts
- tests
- website
- .vitepress
- theme
- guide
- public
- reference/python-api
- backends
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
37 | 48 | | |
38 | 49 | | |
39 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
66 | | - | |
| 66 | + | |
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
| 82 | + | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
| 85 | + | |
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | 83 | | |
146 | 84 | | |
147 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
21 | 53 | | |
22 | 54 | | |
23 | 55 | | |
| |||
0 commit comments