Merge 0.5.11 — FathomDB as a capability, not a sink (internal milestone)

Engine migration to fathomdb 0.8.21 with a legacy-file guard, provenance-keyed
storage (source_id = authenticated client ID) with engine-owned projections,
bounded readers, engine-served log search (#11), per-client erasure (CLI +
admin API), TUI ballast (#24, #27, #28), and the two standing owner decisions.
Internal milestone: no version bump, no v* tag, no PyPI publish.

Author: coreyt

.gitignore

@@ -72,3 +72,4 @@ agents/.aural/
 # Benchmark corpora — third-party datasets, not redistributed by Airlock.
 # See dev/corpora/README.md for provenance and licenses.
 data/
+.coverage

CHANGELOG.md

@@ -7,6 +7,94 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Internal 0.5.11 milestone (not published, 2026-08-07)
+
+The FathomDB 0.8 milestone: the engine migration and its legacy-file guard,
+provenance-keyed storage with engine-owned projections, bounded readers,
+engine-served log search (#11), per-client erasure, and TUI ballast
+(#24, #27, #28). Everything below ships publicly with the next published
+release; the breaking FathomDB change applies only to `db`-extra users.
+
+### Added
+
+- **TUI: failover audit trail (#24).** The model detail pane shows the recent
+  failovers involving that model (timestamp, original → target, reason) from
+  an incremental JSONL tail, and the Overview status line counts failovers in
+  the last 5 minutes — what *actually happened*, alongside the configured
+  chain the models table already shows.
+- **TUI: provider-wide rate-limit escalation (#27).** `impacted_clients` (a
+  live-only value the separate-process TUI cannot compute from its replica)
+  is now part of the admin provider snapshot; the providers table badges the
+  Impacted cell with `⚠ESC` at the escalation threshold, the provider detail
+  pane names the impacted clients and whether escalation is ongoing, and a
+  `provider_escalation` alert rule fires on the event itself.
+- **TUI: Gemini response-mode distribution (#28).** Provider and client detail
+  panes show mode percentages instead of raw counts, with a skew flag when a
+  single mode exceeds 80% of a meaningful sample (≥10 recent responses).
+
+- **Per-client erasure (CLI + admin API).** `airlock admin erase-client
+  <client-id> --confirm <client-id>` and `POST
+  /airlock/admin/clients/{client_id}/erase` remove every FathomDB row whose
+  provenance is that authenticated client id. Loopback-only (like
+  `force_quarantine`), audited with the full `EraseReport` in the
+  `admin_action` record, idempotent, and honest about failure: a partial
+  erasure answers HTTP 409 with the obligation outstanding and is never
+  reported as done. **Scope:** this erases the search/analysis store only —
+  JSONL logs are untouched and governed separately by `AIRLOCK_MAX_LOG_DAYS`;
+  a user-facing deletion obligation requires both.
+- **Log search served by the engine (#11).** `search_request_logs` exposes
+  FathomDB's search over the FTS projections (error text, messages, response
+  text), and the advisor gains a `search_logs` tool on the same seam. The
+  result labels itself honestly: `hybrid` only when dense retrieval could
+  actually contribute; `lexical_only` with the reason otherwise (the normal
+  path here — no embedder is configured); the advisor's JSONL fallback is
+  labelled `substring`. A degraded result is never presented as hybrid, and a
+  hybrid result carries `soft_fallback` when a branch could not contribute.
+
+### ⚠️ Breaking
+
+- **FathomDB migrated to 0.8.x (`fathomdb>=0.8.21,<0.9`).** The 0.3.x store is
+  abandoned with no migration path: Airlock now refuses to open a 0.3.x database
+  file (naming the file and the reason) rather than letting 0.8.x silently adopt
+  it, and the default database filename moved from `airlock.db` to
+  `airlock-fathom.db` so an existing state directory can never be adopted by
+  accident. Old files are left in place; their records remain in the JSONL logs.
+
+### Changed
+
+- The opt-in remote analyzer executor defaults to `claude-sonnet-5`
+  (was `claude-sonnet-4-5`); `AIRLOCK_ANALYZER_REMOTE_MODEL` still overrides.
+  Owner decision — this is a paid path.
+- The Fathom request logger writes 0.8.x dict batches with a mandatory
+  `source_id` on every row: the **authenticated client ID** (`key:<last8>`),
+  stamped by the guardian at pre-call from the validated bearer key and always
+  overwriting any client-supplied value — never the forgeable
+  `X-Airlock-Client` header. Unauthenticated traffic collapses to the
+  `no_client` sentinel, so no write path can produce an unerasable row.
+  `source_id` is the axis per-client erasure will target.
+- RequestLog projections (filterable/rankable fields, FTS over error text,
+  messages, and response text) are declared to the engine via
+  `configure_projections`; Airlock no longer maintains any derived index.
+  No vector projection is declared — no embedder is configured, by design.
+- `api/queries.py` reads through `fathomdb.read` with an explicit `ReadView`
+  (active rows only) and the typed `fathomdb.errors` hierarchy — the
+  `AttributeError`-based capability sniffing and silent empty-list fallbacks
+  are gone.
+- **No datastore read is unbounded anymore.** `get_request_logs` /
+  `get_billing_metrics` default to the shared 50k `DATASTORE_QUERY_LIMIT`
+  (the old default was `limit=1000000` — a limit in name only) and report
+  truncation instead of dropping it: billing metrics carry
+  `truncated`/`limit_hit`, the TUI Overview marks a partial cost sum with
+  `(partial)`, and the advisor's `get_recent_errors` reports the datastore
+  window the same way it reports the JSONL one.
+
+### Removed
+
+- `_ensure_vector_stub_table` (a 0.3.1 write-path workaround; 0.8.x owns its
+  indexes).
+- `search_logs` and its Python-side substring fallback, superseded by the
+  engine's hybrid search (#11).
+
 ## [0.5.10] — 2026-08-05
 
 The first published release since 0.5.8. It carries **two milestones**: the

airlock/admin/erase.py

@@ -0,0 +1,105 @@
+"""Per-client erasure from the FathomDB store (0.5.11 B-2).
+
+``erase_source(client_id)`` removes every row under a provenance together with
+its full-text and secondary-index shadows, and finishes the erasure at rest.
+It is idempotent, so an interrupted obligation can be retried.
+
+**Scoping — this is not full erasure.** It erases a client from the **search
+and analysis store**. The same records exist in JSONL, which ``erase_source``
+does not touch; JSONL retention is governed separately by
+``AIRLOCK_MAX_LOG_DAYS``. A user-facing deletion obligation requires both, and
+the JSONL half is explicitly out of scope for 0.5.11.
+"""
+
+from __future__ import annotations
+
+import datetime
+from typing import Any
+
+
+class EraseIncomplete(Exception):
+    """An erasure half-completed: the obligation is OUTSTANDING.
+
+    Never reported as success — the honest response is "incomplete, retry".
+    Retrying is safe; that is what ``erase_source``'s idempotence buys.
+    Carries the ``admin_action`` audit record for the failed attempt.
+    """
+
+    def __init__(self, record: dict[str, Any]):
+        super().__init__(record.get("error") or "erasure incomplete")
+        self.record = record
+
+
+def _iso_now() -> str:
+    return datetime.datetime.now(datetime.timezone.utc).isoformat()
+
+
+def _report_dict(report: Any) -> dict[str, Any]:
+    """Serialize an ``EraseReport`` — the receipt, not a bare "ok".
+
+    An erasure audit trail that records only that the call was made cannot
+    answer what was actually removed.
+    """
+    return {
+        "source_ref": report.source_ref,
+        "nodes_excised": report.nodes_excised,
+        "edges_excised": report.edges_excised,
+        "projections_invalidated": report.projections_invalidated,
+    }
+
+
+def erase_client(client_id: str, actor: str, *, confirm: Any) -> dict[str, Any]:
+    """Erase every FathomDB row whose provenance is ``client_id``.
+
+    ``confirm`` must repeat the client id — erasure is irreversible and must
+    not be a single mistyped word away.
+
+    Returns an ``admin_action`` record carrying the ``EraseReport``.
+
+    Raises
+    ------
+    ValueError
+        Confirmation mismatch, or the datastore is not enabled here.
+    EraseIncomplete
+        The erasure half-completed; the record says so and retry is safe.
+    """
+    if not client_id:
+        raise ValueError("client_id is required")
+    if confirm != client_id:
+        raise ValueError(
+            "confirmation mismatch: pass the client id again as 'confirm' "
+            "to run this irreversible operation"
+        )
+
+    import airlock.datastore as datastore
+
+    engine = datastore.get_engine()
+    if engine is None:
+        raise ValueError(
+            "FathomDB is not enabled on this proxy (AIRLOCK_ENABLE_FATHOMDB "
+            "unset or the db extra is not installed); nothing to erase here"
+        )
+
+    from fathomdb.errors import ErasureIncompleteError
+
+    base: dict[str, Any] = {
+        "record_type": "admin_action",
+        "timestamp": _iso_now(),
+        "op": "erase_client",
+        "actor": actor,
+        "client_id": client_id,
+        # The scoping constraint, carried into the audit trail itself.
+        "scope_note": "fathomdb store only; JSONL retention is governed by AIRLOCK_MAX_LOG_DAYS",
+    }
+    try:
+        report = engine.erase_source(client_id)
+    except ErasureIncompleteError as exc:
+        raise EraseIncomplete(
+            {
+                **base,
+                "outcome": "incomplete",
+                "error": str(exc),
+                "retry_safe": True,
+            }
+        ) from exc
+    return {**base, "outcome": "complete", "erase_report": _report_dict(report)}

airlock/admin/http.py

@@ -14,6 +14,7 @@
 from typing import Any
 
 import airlock.fast.state as _state
+from airlock.admin.erase import EraseIncomplete, erase_client
 from airlock.admin.policy import LOOPBACK_HOSTS, Principal, admin_enabled, decide
 from airlock.callbacks.enterprise_logger import write_admin_action_record
 from airlock.fast.settings import get_settings
@@ -40,6 +41,9 @@ def _view_providers() -> dict:
             "cooldown_remaining": round(ps.cooldown_remaining(), 1) if ps else 0.0,
             "half_open": ps._half_open_probe if ps else False,
             "last_reason": ps.last_reason if ps else "",
+            # Live-only state (rate_limit_events with CC-6 floors) — the
+            # separate-process TUI cannot compute this from its replica (#27).
+            "impacted_clients": sorted(ps.impacted_clients()) if ps else [],
             "remaining_tokens": rl.remaining_tokens if rl else None,
             "limit_tokens": rl.limit_tokens if rl else None,
             "remaining_requests": rl.remaining_requests if rl else None,
@@ -146,6 +150,13 @@ def _match_route(method: str, path: str):
             False,
             lambda p, b, a: _state.store.clear_client_backoff(client, actor=a),
         )
+    if method == "POST" and len(seg) == 3 and seg[0] == "clients" and seg[2] == "erase":
+        client = seg[1]
+        return (
+            "admin:erase_client",
+            True,  # loopback-only (operator) — destructive, like force_quarantine
+            lambda p, b, a: erase_client(client, a, confirm=b.get("confirm")),
+        )
     if (
         method == "POST"
         and len(seg) == 3
@@ -192,6 +203,11 @@ def handle_admin_request(
             write_admin_action_record(result)
     except ValueError as exc:
         return 400, {"error": str(exc)}, {}
+    except EraseIncomplete as exc:
+        # A partial erasure is never reported as complete. Audit the attempt,
+        # answer 409 with the obligation outstanding; retrying is safe.
+        write_admin_action_record(exc.record)
+        return 409, dict(exc.record), {}
     except Exception:  # noqa: BLE001 — the perimeter must never raise (CC-10)
         return 500, {"error": "internal error"}, {}
     return 200, result, {}

airlock/advisor/tools.py

@@ -23,13 +23,15 @@
 from pathlib import Path
 from typing import Any, Callable
 
-from airlock.api.queries import get_request_logs
+from airlock.api.queries import (
+    DATASTORE_QUERY_LIMIT,
+    get_request_logs,
+    node_properties,
+    search_request_logs,
+)
 from airlock.fast.state import StateStore
 from airlock.log_query import LogPage, LogQuery, query_logs
 
-#: Upper bound on rows pulled from the datastore in one advisor query.
-DATASTORE_QUERY_LIMIT = 50_000
-
 logger = logging.getLogger("airlock.advisor.tools")
 
 
@@ -145,10 +147,11 @@ def get_recent_errors(log_dir: str, days: int = 2) -> dict:
 
     truncated: dict[str, Any] = {"truncated": False, "limit_hit": None}
     if engine is not None:
-        # Was limit=1000000 — a limit in name only, and every row was
-        # materialized before filtering.
+        # Bounded read (was limit=1000000 — a limit in name only). The failure
+        # filter stays Python-side: the engine's json-path predicate allowlist
+        # is fixed and does not cover $.success.
         nodes = get_request_logs(engine, limit=DATASTORE_QUERY_LIMIT)
-        records = [n.properties if hasattr(n, "properties") else n for n in nodes]
+        records = [node_properties(n) for n in nodes]
         failures = [
             r
             for r in records
@@ -200,6 +203,48 @@ def get_recent_errors(log_dir: str, days: int = 2) -> dict:
     }
 
 
+# ---------------------------------------------------------------------------
+# Tool 2b: search_logs
+# ---------------------------------------------------------------------------
+
+
+def search_logs(log_dir: str, query: str, limit: int = 20, days: int = 7) -> dict:
+    """Search request logs (#11): engine search when available, JSONL scan otherwise.
+
+    The engine path reports its ``mode`` honestly — ``hybrid`` only when dense
+    retrieval actually contributed alongside lexical; ``lexical_only`` with the
+    reason otherwise. The JSONL fallback is a bounded substring scan and says
+    so: ``substring`` is not search-engine ranking and must not look like it.
+    """
+    engine = None
+    try:
+        import airlock.datastore
+
+        engine = airlock.datastore.get_engine()
+    except Exception:
+        engine = None
+
+    if engine is not None:
+        out = search_request_logs(engine, query, limit=limit)
+        out["backend"] = "fathomdb"
+        return out
+
+    needle = query.lower()
+
+    def _matches(record: dict[str, Any]) -> bool:
+        return needle in json.dumps(record, default=str).lower()
+
+    page = _load_page(log_dir, days=days, predicate=_matches)
+    return {
+        "backend": "jsonl",
+        "mode": "substring",
+        "degraded_reason": "datastore disabled; bounded JSONL substring scan",
+        "soft_fallback": None,
+        "results": page.records[-limit:],
+        "window": _truncation(page),
+    }
+
+
 # ---------------------------------------------------------------------------
 # Tool 3: get_analysis_report
 # ---------------------------------------------------------------------------
@@ -467,6 +512,30 @@ def get_knobs(log_dir: str) -> dict:
             "description": "Get recent error records grouped by model, client, and error type",
         },
     ),
+    "search_logs": (
+        search_logs,
+        {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "Search text (matched against error text, messages, and response text)",
+                },
+                "limit": {
+                    "type": "integer",
+                    "description": "Maximum results to return",
+                    "default": 20,
+                },
+                "days": {
+                    "type": "integer",
+                    "description": "JSONL fallback window in days (ignored on the datastore path)",
+                    "default": 7,
+                },
+            },
+            "required": ["query"],
+            "description": "Search request logs. Result carries mode: hybrid, lexical_only (with reason), or substring (JSONL fallback)",
+        },
+    ),
     "get_analysis_report": (
         get_analysis_report,
         {