fix(layers): fetch the most recent drawers for L1 wake-up

PR #1630 fixed the L1 wake-up ordering by adding filed_at as a secondary
sort key, and documented what it could not fix:

  "The fetch still scans up to MAX_SCAN=2000 drawers in collection
  (insertion) order, so on a very large unscoped wing the most-recent
  drawers may fall outside the cap before sorting... A SQL-side
  most-recent-N by filed_at fetch would fix the unscoped case but is a
  larger change left for a follow-up."

This is that follow-up. On a 149k-drawer palace the sort was correct and
the input was not: the 2000 drawers Layer1 scored were the oldest
backfill slice, so wake-up permanently opened on the first files ever
mined and never on this week's sessions.

Backend capability rather than a pgvector special case:

- BaseCollection.get_recent(limit, where, order_field, include) returns
  up to limit records newest-first on an ISO-8601 metadata field.
  The ABC default pages through get() and sorts the window locally,
  which is exactly what Layer1 did inline, so every backend that does
  not override it behaves as before.
- PgVectorCollection overrides it with ORDER BY metadata->>%s DESC
  NULLS LAST, id pushed into the scan, and PgVectorBackend advertises
  the supports_recency_order capability token. That is exact at any
  table size. Filters that pgvector cannot push down exactly keep the
  existing local post-filter path.
- EmbeddingCollection forwards get_recent explicitly. Without the
  forwarder, MRO would resolve the concrete ABC default on the wrapper
  and shadow the inner backend's pushdown (the invariant
  test_wrapper_forwards_all_concrete_basecollection_methods guards).
- Layer1._fetch_candidates uses the capability and falls back to the
  previous paged scan when a collection predates get_recent or the
  backend errors, so wake-up degrades instead of failing.

recency_sort_key is shared so every local sort orders identically:
records missing the field, holding an empty string, or holding a
non-string sort last instead of raising on a str/None comparison.

No ordering semantics change for Layer1 itself. importance stays the
primary key and filed_at the tiebreak; only the candidate window
changes, from "the first 2000 rows the backend hands back" to "the 2000
most recently filed".

Measured on the 149k-drawer palace this was written for: wake-up now
leads with the newest sessions and renders in 0.72s.

Tests: base default (ordering, missing/odd timestamps, window cap,
where passthrough, dict-shaped get), pgvector pushdown (SQL text and
bind order, filter pushdown, local-filter fallback, include, zero
limit, custom order field), and Layer1 (capability used, wing filter
forwarded, fallback when the capability is missing or raises).

Author: ATKabli

CHANGELOG.md

@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 - **`sweep` books a failed `stat` as a failure again.** The non-regular-file gate added in 3.7.1 reads `stat.S_ISREG(f.stat().st_mode)` inside a `try`, and its `except OSError` printed `SKIP` and moved on. A dangling symlink, a symlink loop and a file unlinked between `rglob` and the gate all raise there, and before the gate existed every one of them reached `sweep()` and was booked in `failures` — so `sweep` went from reporting a transcript it could not read to reporting success. A failed probe is now an error, not a benign file type: it is logged, printed as `WARNING`, and appended to `failures`, while a probe that succeeds and says "not regular" still skips silently. (#2221)
 - **`mempalace init` no longer tracebacks on a directory it cannot enter.** `_parse_gradle`'s `is_file()` gate sat in front of the `try` that the parser's own `except OSError` provides, so a manifest under a directory with `r` but no `x` raised `PermissionError` out of a call that used to answer "no manifest name". The gate moved inside that `try`, and `_collect_manifest_names` stats through `os.path.isfile`, which reports rather than raises. (#2221)
 - **`split` no longer blocks on a FIFO at its own output name, nor writes through a broken link.** The type gate in `main` covers the files the glob listed; `split_file` builds its output names itself, so a pre-existing named pipe at one of them wedged `write_text` in the kernel waiting for a reader. The check asks about the link itself rather than its target, because a dangling symlink at an output name reads as "nothing there" and `write_text` would create the target — landing a chunk outside the output directory. Output names that are anything but a regular file are now skipped with a `SKIP` line. (#2221)
+- **Layer 1 wake-up fetches the most recent drawers instead of an arbitrary scan window.** `BaseCollection.get_recent()` returns the newest N by `filed_at`; pgvector overrides it with an `ORDER BY ... DESC LIMIT n` pushdown (capability token `supports_recency_order`), and backends without pushdown keep the previous scan-and-sort behavior. On palaces larger than the 2,000-drawer scan cap, wake-up no longer leads with the oldest backfill. (#1630)
 
 ---
 

mempalace/backends/base.py

@@ -354,6 +354,30 @@ class LexicalResult:
     hits: list[LexicalHit]
 
 
+def recency_sort_key(meta: Optional[dict], order_field: str = "filed_at") -> tuple[int, str]:
+    """Sort key for newest-first ordering on an ISO-8601 metadata field.
+
+    Returns ``(1, value)`` for a usable timestamp string and ``(0, "")``
+    otherwise, so that with ``reverse=True`` records missing the field sort
+    last instead of raising on a str/None comparison. Backends that implement
+    :meth:`BaseCollection.get_recent` with a local sort MUST use this key so
+    every backend orders identically.
+    """
+    value = (meta or {}).get(order_field)
+    if not isinstance(value, str) or not value:
+        return (0, "")
+    return (1, value)
+
+
+def _recency_order(metadatas: list[dict], order_field: str) -> list[int]:
+    """Indices into ``metadatas``, newest first, missing timestamps last."""
+    return sorted(
+        range(len(metadatas)),
+        key=lambda i: recency_sort_key(metadatas[i], order_field),
+        reverse=True,
+    )
+
+
 # ---------------------------------------------------------------------------
 # Collection contract
 # ---------------------------------------------------------------------------
@@ -504,6 +528,74 @@ def get_all_metadata(self, where: Optional[dict] = None) -> list[dict]:
             offset += len(batch_meta)
         return all_meta
 
+    def get_recent(
+        self,
+        *,
+        limit: int,
+        where: Optional[dict] = None,
+        order_field: str = "filed_at",
+        include: Optional[list[str]] = None,
+    ) -> GetResult:
+        """Return up to ``limit`` records, newest first by ``order_field``.
+
+        ``order_field`` names a metadata key holding an ISO-8601 timestamp
+        string (``filed_at`` for drawers). Ordering is descending on that
+        string, which is chronological for ISO-8601. Records whose value is
+        missing, empty, or not a string sort last.
+
+        The default implementation pages through :meth:`get` in storage order
+        up to ``limit`` records and sorts that window locally. That is exact
+        for a collection no larger than ``limit`` and *approximate* above it:
+        the window is whatever the backend hands back first, so the genuinely
+        newest records can fall outside it (issue #1630's known limitation for
+        Layer 1 wake-up). Backends able to push the ordering into storage MUST
+        override this and advertise the ``supports_recency_order`` capability
+        token, which promises exactness at any collection size.
+
+        Callers that need the guarantee should check the token rather than
+        assume it; the default is always available so no backend breaks.
+        """
+        if limit <= 0:
+            return GetResult.empty()
+        include = ["documents", "metadatas"] if include is None else list(include)
+
+        ids: list[str] = []
+        documents: list[str] = []
+        metadatas: list[dict] = []
+        offset = 0
+        fetched = 0
+        page_size = min(500, limit)
+        while fetched < limit:
+            kwargs: dict = {"include": include, "limit": page_size, "offset": offset}
+            if where:
+                kwargs["where"] = where
+            batch = self.get(**kwargs)
+            batch_ids = list(batch.get("ids") or [])
+            batch_docs = list(batch.get("documents") or [])
+            batch_metas = list(batch.get("metadatas") or [])
+            page_len = max(len(batch_ids), len(batch_docs), len(batch_metas))
+            if not page_len:
+                break
+            # Pad the projections the caller did not request so the three
+            # lists stay index-aligned for the sort below.
+            ids.extend(batch_ids or [""] * page_len)
+            documents.extend(batch_docs or [""] * page_len)
+            metadatas.extend(batch_metas or [{}] * page_len)
+            offset += page_len
+            fetched += page_len
+            if page_len < page_size:
+                break
+
+        n = min(len(ids), len(documents), len(metadatas))
+        ids, documents, metadatas = ids[:n], documents[:n], metadatas[:n]
+        order = _recency_order(metadatas, order_field)[:limit]
+        return GetResult(
+            ids=[ids[i] for i in order],
+            documents=[documents[i] for i in order],
+            metadatas=[metadatas[i] for i in order],
+            embeddings=None,
+        )
+
     def facet_counts(
         self,
         field: str,

mempalace/backends/embedding_wrapper.py

@@ -167,6 +167,21 @@ def health(self):
     def lexical_search(self, *, query: str, n_results: int = 10, where: Optional[dict] = None):
         return self._inner.lexical_search(query=query, n_results=n_results, where=where)
 
+    def get_recent(
+        self,
+        *,
+        limit: int,
+        where: Optional[dict] = None,
+        order_field: str = "filed_at",
+        include: Optional[list[str]] = None,
+    ):
+        # Concrete on ``BaseCollection`` (the scan-and-sort default), so MRO
+        # would resolve it here and shadow a backend that pushes the ordering
+        # into storage. Forward explicitly.
+        return self._inner.get_recent(
+            limit=limit, where=where, order_field=order_field, include=include
+        )
+
     def facet_counts(
         self, field: str, where: Optional[dict] = None, limit: int = 1000
     ) -> dict[str, int]:

mempalace/backends/pgvector.py

@@ -678,6 +678,7 @@ def scroll_rows(
         with_document: bool = True,
         limit: Optional[int] = None,
         offset: Optional[int] = None,
+        order_field: Optional[str] = None,
     ) -> list[dict]:
         qi = _quote_identifier(table)
         params: list = []
@@ -695,7 +696,21 @@ def scroll_rows(
         # primary key gives OFFSET a stable order (an unordered scan may skip
         # or repeat rows across pages); callers that scroll the whole table
         # pass neither bound, leaving their SQL unchanged.
-        if limit is not None or offset:
+        if order_field is not None:
+            # Newest-first on an ISO-8601 metadata field. ISO-8601 sorts
+            # chronologically as text, so ``metadata->>field DESC`` needs no
+            # timestamp cast (a cast would also fail hard on one malformed
+            # value). NULLS LAST keeps records without the field at the end;
+            # ``id`` breaks ties so the order is total and stable.
+            params.append(order_field)
+            sql += " ORDER BY metadata->>%s DESC NULLS LAST, id"
+            if limit is not None:
+                params.append(int(limit))
+                sql += " LIMIT %s"
+            if offset:
+                params.append(int(offset))
+                sql += " OFFSET %s"
+        elif limit is not None or offset:
             sql += " ORDER BY id"
             if limit is not None:
                 params.append(int(limit))
@@ -893,6 +908,7 @@ def _scroll(
         with_document=True,
         limit=None,
         offset=None,
+        order_field=None,
     ) -> list[dict]:
         self._ensure_open()
         if not self._table_exists():
@@ -906,6 +922,7 @@ def _scroll(
             with_document=with_document,
             limit=limit,
             offset=offset,
+            order_field=order_field,
         )
 
     def get_all_metadata(self, where=None) -> list[dict]:
@@ -1194,6 +1211,42 @@ def get(
             embeddings=[row["embedding"] or [] for row in rows] if spec.embeddings else None,
         )
 
+    def get_recent(self, *, limit, where=None, order_field="filed_at", include=None):
+        """Newest-first fetch with the ordering pushed into SQL.
+
+        The base implementation scans a window in storage order and sorts it
+        locally, so on a collection larger than ``limit`` the genuinely newest
+        records can be missing from the window entirely (#1630). Postgres can
+        do the whole thing: ``ORDER BY metadata->>'filed_at' DESC ... LIMIT n``
+        is exact at any table size, which is why this backend advertises
+        ``supports_recency_order``.
+
+        Filters that ``metadata @> ...`` cannot express exactly fall back to
+        the local post-filter path (same correctness contract as ``get``), and
+        there the SQL order is still applied before the filter, so the result
+        is the newest ``limit`` rows *that match* only when the pushdown was
+        exact. The inexact case re-sorts locally after filtering.
+        """
+        if limit is not None and limit <= 0:
+            return GetResult.empty()
+        _validate_where(where)
+        spec = _IncludeSpec.resolve(include, default_distances=False)
+        local_filter = _requires_local_filter(where)
+        rows = self._scroll(
+            where=None if local_filter else where,
+            with_embedding=spec.embeddings,
+            limit=None if local_filter else limit,
+            order_field=order_field,
+        )
+        if local_filter:
+            rows = [row for row in rows if _matches_where(row["metadata"], where)][:limit]
+        return GetResult(
+            ids=[row["id"] for row in rows],
+            documents=[row["document"] for row in rows] if spec.documents else [],
+            metadatas=[row["metadata"] for row in rows] if spec.metadatas else [],
+            embeddings=[row["embedding"] or [] for row in rows] if spec.embeddings else None,
+        )
+
     def delete(self, *, ids=None, where=None):
         _validate_where(where)
         if not self._table_exists():
@@ -1336,6 +1389,7 @@ class PgVectorBackend(BaseBackend):
             "supports_metadata_filters",
             "supports_lexical_search",
             "supports_metadata_facets",
+            "supports_recency_order",
             "supports_namespace_isolation",
             "supports_server_side_indexes",
             "server_mode",

mempalace/layers.py

@@ -87,28 +87,47 @@ class Layer1:
 
     MAX_DRAWERS = 15  # at most 15 moments in wake-up
     MAX_CHARS = 3200  # hard cap on total L1 text (~800 tokens)
-    MAX_SCAN = 2000  # don't scan more than this for L1 generation
+    MAX_SCAN = 2000  # size of the candidate window pulled for L1 generation
 
     def __init__(self, palace_path: str = None, wing: str = None):
         cfg = MempalaceConfig()
         self.palace_path = palace_path or cfg.palace_path
         self.wing = wing
 
-    def generate(self) -> str:
-        """Pull top drawers from ChromaDB and format as compact L1 text."""
-        try:
-            col = _get_collection(self.palace_path, create=False)
-        except Exception:
-            return "## L1 — No palace found. Run: mempalace mine <dir>"
+    def _fetch_candidates(self, col) -> tuple[list, list]:
+        """Fetch the L1 candidate window: the MAX_SCAN most recently filed drawers.
 
-        # Fetch all drawers in batches to avoid SQLite variable limit (~999)
+        Uses the backend's ``get_recent`` capability, which pushes
+        ``ORDER BY filed_at DESC LIMIT n`` into storage where the backend can
+        (pgvector today) and otherwise falls back to the scan-then-sort default
+        in ``BaseCollection``. That default is what this method used to do
+        inline, so backends without pushdown behave exactly as before.
+
+        Third-party collections predating ``get_recent`` (or any backend error)
+        degrade to the inline paged scan below rather than failing wake-up.
+        """
+        where = {"wing": self.wing} if self.wing else None
+        getter = getattr(col, "get_recent", None)
+        if getter is not None:
+            try:
+                result = getter(
+                    limit=self.MAX_SCAN,
+                    where=where,
+                    order_field="filed_at",
+                    include=["documents", "metadatas"],
+                )
+                return list(result.documents or []), list(result.metadatas or [])
+            except Exception:
+                pass  # capability missing or backend hiccup — page it manually
+
+        # Fetch in batches to avoid SQLite variable limit (~999)
         _BATCH = 500
         docs, metas = [], []
         offset = 0
         while True:
             kwargs = {"include": ["documents", "metadatas"], "limit": _BATCH, "offset": offset}
-            if self.wing:
-                kwargs["where"] = {"wing": self.wing}
+            if where:
+                kwargs["where"] = where
             try:
                 batch = col.get(**kwargs)
             except Exception:
@@ -122,6 +141,16 @@ def generate(self) -> str:
             offset += len(batch_docs)
             if len(batch_docs) < _BATCH or len(docs) >= self.MAX_SCAN:
                 break
+        return docs, metas
+
+    def generate(self) -> str:
+        """Pull top drawers from the palace and format as compact L1 text."""
+        try:
+            col = _get_collection(self.palace_path, create=False)
+        except Exception:
+            return "## L1 — No palace found. Run: mempalace mine <dir>"
+
+        docs, metas = self._fetch_candidates(col)
 
         if not docs:
             return "## L1 — No memories yet."
@@ -136,6 +165,10 @@ def generate(self) -> str:
         # newest first. This keeps importance as the primary key for the day a
         # scoring pass populates it, while making the "recent filing" half of
         # the promise true today with data we already have.
+        # The candidate window this sorts is now the MAX_SCAN *most recently
+        # filed* drawers rather than the first MAX_SCAN the backend happened to
+        # hand back, so on a palace larger than MAX_SCAN the newest drawers are
+        # actually in the running (#1630's known limitation).
         scored = []
         for doc, meta in zip(docs, metas):
             meta = meta or {}