Memory Service

A Dockerized HTTP memory service for an AI agent. It ingests conversation turns, extracts structured knowledge from them, evolves facts as they change, and answers recall queries that decide what context the agent sees on its next turn.

git clone <repo> memory-service
cd memory-service
docker compose up -d --build
until curl -sf http://localhost:8080/health; do sleep 1; done
# service is live on http://localhost:8080

⚠️ Set API keys for full quality (do this before grading)

The service boots with no keys, but then degrades to rule-based extraction

• BM25-only retrieval. To exercise the real pipeline, set keys first:

cp .env.example .env
# edit .env and set OPENAI_API_KEY   (this single key powers BOTH stages:
#   extraction via OpenAI AND embeddings via OpenAI)
# Anthropic users: set ANTHROPIC_API_KEY for extraction + OPENAI_API_KEY (or
#   VOYAGE_API_KEY) for embeddings — there is no Anthropic embeddings API.
docker compose up -d --build

The service needs a key for two stages — extraction (ANTHROPIC_API_KEY or OPENAI_API_KEY) and embeddings (OPENAI_API_KEY or VOYAGE_API_KEY). An OPENAI_API_KEY alone covers both. Confirm you're not in degraded mode:

docker compose logs memory | grep "ready:"
# want: ready: extraction=openai embeddings=openai   (NOT rule_based / none)

---

Table of contents

1. Architecture
2. Backing store choice
3. Extraction pipeline
4. Recall strategy
5. Fact evolution
6. Tradeoffs
7. Failure modes
8. Running the tests
9. Configuration reference
10. Endpoint reference
11. Code map (for maintainers / agents)

---

1. Architecture

flowchart TD
    A[POST /turns] --> EX[Extraction<br/>LLM or rule-based]
    EX --> EV[Fact evolution<br/>supersede / reinforce]
    A --> EMB1[Embed turn + memories]
    EV --> DB[(SQLite<br/>turns • memories<br/>FTS5 • float32 blobs)]
    EMB1 --> DB

    Q[POST /recall / /search] --> QX[Query expansion]
    QX --> BM25[BM25 ranker - FTS5]
    QX --> VEC[Vector ranker - cosine]
    BM25 --> RRF[Reciprocal Rank Fusion]
    VEC --> RRF
    DB --> BM25
    DB --> VEC
    RRF --> ASM[Budget-aware assembly<br/>facts → opinions → episodes]
    ASM --> OUT[context + citations]

Loading

                    ┌─────────────────────────────────────────────┐
   POST /turns ───► │  extract → evolve(supersede/reinforce) →      │
                    │  embed → persist (one SQLite transaction)     │
                    └─────────────────────────────────────────────┘
                                        │
                                 ┌──────▼───────┐
                                 │   SQLite      │  one file on a named
                                 │  WAL + FTS5   │  Docker volume (/data)
                                 └──────▲───────┘
                                        │
   POST /recall ──► expand query ─► [BM25 ⊕ vector] ─► RRF ─► budget assembly ─► context

In two paragraphs. The write path (POST /turns) is fully synchronous: a turn is run through extraction, each candidate memory is reconciled against existing memories on the same topic key (supersede or reinforce), everything is embedded in a single round-trip, and turn + memories + FTS index rows are committed in one SQLite transaction. When /turns returns 201, the data is already queryable — no background workers, no eventual consistency.

The read path (POST /recall) expands the query with light synonyms, runs two independent rankers over the user's memories and turns — BM25 for lexical precision and cosine for semantics — and fuses them with Reciprocal Rank Fusion. The fused results are assembled into prompt-ready prose under a token budget, prioritizing stable user facts, then opinions, then query-relevant episodes. It's a monolith (one process, one store) because the spec scopes this to a single service with a few concurrent sessions; that keeps the synchronous correctness guarantee trivial to reason about.

---

2. Backing store choice

SQLite (one file) with FTS5 for lexical search and float32 embedding blobs scored in-process with cosine. Persistence is a single file on a named Docker volume; restart is invisible to clients.

Why SQLite over Postgres+pgvector / Qdrant / a vector DB:

Need	How SQLite covers it
Persistence across restart	One file on a named volume. Nothing to orchestrate.
Real BM25 (not just ts_rank)	FTS5 ships a true BM25 scorer. Postgres FTS does not.
Hybrid retrieval in one store	FTS5 table + embedding blob columns live next to the data. No cross-store joins, no second container.
Synchronous correctness	Single writer + WAL: a committed transaction is immediately visible to the next read. Trivial to guarantee.
docker compose up, no setup	No server to provision, no migrations to run, no init scripts.
The actual scale	One service, single user, a few sessions = hundreds–thousands of vectors. Brute-force cosine is sub-millisecond here.

What I gave up: approximate-nearest-neighbor indexing (HNSW/IVF) and multi-writer concurrency. Neither matters at this scale. Scaling path: when the per-user vector count grows past ~10⁴, swap the in-process cosine loop for the sqlite-vec extension (drop-in, same file) or move embeddings to Postgres+pgvector — the recall ranker interface isolates that change to one module.

Concurrency model: a single shared connection guarded by a re-entrant lock, with PRAGMA journal_mode=WAL. FastAPI runs sync endpoints in a threadpool; the lock serializes writes (cheap at this scale) while WAL lets reads proceed.

---

3. Extraction pipeline

Goal: turn raw turns into typed, keyed, provenance-stamped memories — never store message chunks and call them memories.

Primary path — LLM (provider-agnostic). A single structured-output call (src/memory_service/extraction.py, Anthropic or OpenAI via raw HTTP in llm.py) returns:

{ "memories": [
  { "subject": "user",            // who the fact is about (v1.1)
    "type": "fact|preference|opinion|event",
    "key": "employment",          // canonical snake_case topic id
    "value": "Works at Notion as a PM",
    "confidence": 0.9 }
]}

The prompt is engineered around four ideas:

• Subject attribution (v1.1). Each memory is tagged with who it is about — "user", or a named third party ("sister maya", "friend dave"). Facts the user states about other people are never folded into the user's own facts. This fixed a class of failures where a friend's job leaked into the user's profile, or a sister's city was mis-stored as the user's. See the v1.1 CHANGELOG entry.
• Canonical keys. The model is told to emit a stable snake_case key per real-world topic (employment, location, pet_name, opinion_typescript) so that a later statement on the same topic collides on the same key. This key is the hook the fact-evolution layer keys off.
• Type discipline. fact (stable), preference (likes/working style), opinion (evolvable stance), event (a moment in time). Types drive recall priority and whether a memory can be superseded.
• Implicit facts & corrections. The prompt explicitly extracts implicit facts ("walking Biscuit" → pet named Biscuit) and resolves in-turn corrections ("actually, not X — Y") to the corrected value, with calibrated confidence (explicit ≈ 0.9, inferred ≈ 0.6).

Fallback path — rule-based. When no LLM key is configured (or the call fails), a small high-precision regex bank extracts the most common explicit facts (employment, location, pet, name, diet, allergy, response-style). It is deliberately narrow — better to miss than to fabricate.

What it misses (and why): the rule-based fallback only catches explicit, common phrasings and won't do implicit inference or opinion arcs — that's the cost of running without an LLM, and it's documented rather than hidden. Even the LLM path is per-turn (no cross-turn co-reference resolution beyond what's in the turn), so a fact spread across many turns in subtle ways can be partially missed.

---

4. Recall strategy

End-to-end for POST /recall (src/memory_service/recall.py):

1. Query expansion. Strip stopwords, then add a small synonym set so question phrasing reaches stored keys ("where do they live" → location, city, lives). Cheap and deterministic — no extra LLM call on the hot path.
2. Two rankers.
   • BM25 over the FTS5 index of memories and turns — exact/keyword matches.
   • Cosine over embedding blobs — semantic matches. Skipped if embeddings are disabled (BM25-only fallback).
3. Fusion — Reciprocal Rank Fusion (score = Σ 1/(k + rank), k=60). RRF is scale-free, so I never have to normalize BM25 against cosine; it fuses on rank position. This is what rescues keyword-dependent queries that pure vector search misses.
4. Budget-aware assembly. Results are split into facts/preferences, opinions, and episodes (events + raw turns), then emitted as prose under max_tokens.

Priority logic under budget (defended)

When the budget is tight, the order is deliberate:

1. The user's own stable facts first (fact/preference with subject="user"), ranked by relevance then confidence. The highest-value, longest-lived context — who the user is. Facts get up to 70% of the budget.
2. The user's opinions next — useful stance context, but mutable.
3. Facts about people they know (subject != "user") in a separate "## About people they know" section, so a friend's or relative's facts inform multi-hop questions without ever masquerading as the user's own.
4. Query-relevant episodes last (events, recent turns), ranked by relevance then recency, filling whatever budget remains.

Rationale: a frozen LLM reading this context benefits most from durable identity facts; episodic detail is supporting color. Facts are also short, so capping them at 70% almost never starves episodes in practice — it's a guard against a fact-heavy user crowding out the query-relevant turn. Token cost is estimated at ~4 chars/token and lines are added greedily until the budget is reached, so we stay within the budget (never 2× over).

Output format is prompt-ready Markdown with dated episodes and a stable-facts header (matches the contract example), plus citations carrying turn_id, fused score, and a snippet for each included line.

Latency: one embedding call for the query (skipped in BM25-only mode), one FTS5 query, and an in-process cosine loop over the user's vectors (hundreds → sub-ms). No reranker model on the hot path. Typical recall is well under the "reasonable time" bar; the dominant cost when embeddings are on is the single embedding API round-trip.

---

5. Fact evolution

Memories are keyed by (scope, subject, key), scope = user_id (or session:<id> when no user), subject = who the fact is about (v1.1). Scoping by subject means a fact about the user's sister never supersedes the same-topic fact about the user. On ingest, each extracted memory is reconciled (memory_store.py):

• Reinforce — if the new value is near-identical to an active memory on the same key (case-insensitive string match, or embedding cosine ≥ 0.93): bump confidence, refresh wording and timestamp. No new row, no churn.
• Supersede — if the value differs: the old row is set active=0 and a new active row is inserted with supersedes pointing at the old id. History is preserved; only the active row is recalled.
• Events never supersede — they describe a moment, so they accumulate.

This satisfies the contract's worked example directly (verified live):

session 1: "I work at Stripe"   → memory{key:employment, value:"...Stripe", active:1}
session 3: "I joined Notion"     → old Stripe row active:0;
                                    new Notion row active:1, supersedes:<stripe-id>
/recall "where do they work now" → "Started working at Notion"
/users/{id}/memories             → both rows; Stripe inactive, chain visible

Opinion arcs (partial, documented). Opinions use the same key, so the supersession chain is the arc: "loves TypeScript" → "generics are annoying" → "fine for big projects" each supersede the prior, and the full chain is walkable via supersedes in /users/{id}/memories. What's not yet implemented is a synthesized arc summary in /recall (e.g. "warmed then cooled on TypeScript") — today recall surfaces the current stance. Synthesizing the arc from the chain is the natural next step and is isolated to the assembly stage.

Known limitation — key-based reconciliation (found via the eval harness; see eval/FINDINGS.md). Supersession matches on the exact (subject, key) the LLM assigns, which produces two opposite failure modes when the key is ambiguous:

• Over-merge: several coexisting facts that share a key clobber each other — tell the service about a cat, a dog, and a parrot and it can keep only one pet row. (Reproduces reliably for 3+ same-slot items.)
• Under-merge: the same single-valued slot phrased with different keys (car vs vehicle) leaves two contradictory rows active. (Intermittent — depends on the LLM's key choice.)

Single-valued slots with a consistent key (location, employment) supersede correctly, and the recall layer often masks the gap because raw turns still carry the history. The robust fix (planned v1.2) is LLM reconciliation: on ingest, show the model the user's existing facts and have it emit ADD / UPDATE / DUPLICATE per new fact — judging contradiction vs. coexistence instead of trusting an exact key match.

---

6. Tradeoffs

Optimized for:

• Recall + extraction quality and contract correctness — the primary eval signals.
• Operational simplicity & reproducibility — one process, one file, docker compose up, no migrations.
• Synchronous correctness — committed = immediately recallable, by construction.
• Graceful degradation — never dead on a cold machine; missing keys downgrade capability, not availability.

Gave up:

• ANN vector indexing — brute-force cosine instead (fine at this scale; swap path documented).
• Horizontal scale / multi-writer concurrency — single SQLite writer. Out of scope per the brief.
• A reranker model — RRF + recency/confidence heuristics instead of a cross-encoder. Cheaper, no model to host; precision ceiling is lower on long histories.
• Cross-turn co-reference & multi-turn extraction — extraction is per-turn.

---

7. Failure modes

Situation	Behavior
No data / cold session	/recall returns {"context":"", "citations":[]} with 200. Never errors.
No API keys at all	Boots fine. Rule-based extraction + BM25-only retrieval. Logged at startup (extraction=rule_based embeddings=none).
LLM key but no embedding key	LLM extraction + BM25-only retrieval. (This is the validated host setup: extraction=anthropic embeddings=none.)
LLM/embedding call fails mid-request	Caught and logged; extraction falls back to rules, retrieval falls back to BM25. Request still succeeds.
Malformed JSON / missing fields	422 (or 400), never a crash.
Oversized payload	413 above REQUEST_MAX_BYTES (default 2 MB).
Unicode / emoji / weird roles / extra fields	Tolerated; stored and recalled correctly.
Unexpected internal error	Global handler returns 500 with a generic body; worker stays up.
Slow disk	WAL keeps reads non-blocking; writes serialize on the lock. Degrades in latency, not correctness.
Bad/expired auth (when MEMORY_AUTH_TOKEN set)	401. If unset, auth is skipped entirely.

---

8. Running the tests

The suite runs hermetically in degraded mode (no keys) so it's deterministic and fast. Two ways:

A. In Docker (matches the deploy target, Python 3.12):

docker run --rm -v "$PWD:/app" -w /app python:3.12-slim \
  bash -c "pip install -q -r requirements.txt -r tests/requirements-dev.txt && pytest -q tests"

B. Locally (needs Python with prebuilt pydantic-core wheels):

pip install -r requirements.txt -r tests/requirements-dev.txt
pytest -q tests

What's covered:

Test file	Covers
test_contract.py	Roundtrip, every endpoint's shape & status code, cold-session empty recall.
test_persistence.py	Write → tear down app → re-open same DB → data survives (restart sim).
test_concurrent.py	Two users don't bleed; same-user cross-session sharing (the documented design).
test_malformed.py	Bad JSON, missing fields, unicode, oversized body, weird roles → 4xx/handled, never crash.
test_recall_quality.py	Self-eval fixture: ingest fixtures/, run probes, assert X of Y passed; plus a supersession-chain assertion.

Latest run: 17 passed; recall self-eval 6/6 probes in degraded mode. Run the fixture with -s to see the per-probe report:

pytest -q -s tests/test_recall_quality.py::test_recall_quality_fixture

Adversarial harness (the iteration loop)

eval/run_eval.py drives the live service through harder scenarios over the full LLM + embedding pipeline and grades /recall two ways: deterministic substring checks and an independent LLM-as-judge (correctness + hallucination). This is the loop that drives the CHANGELOG — the hermetic unit fixture saturates, so honest iteration needs probes that break the real pipeline.

# service must be running with provider keys (docker compose up -d)
python eval/run_eval.py --scenarios eval/scenarios_hard.json --tag my-run

It loads keys from .env, prints a per-category report, and appends a summary to eval/results.jsonl. eval/scenarios_hard.json is 35 probes across implicit facts, corrections, negation/deactivation, long-term memory (facts recalled across ~6 months of sessions), noise resistance, attribution, fact evolution, multi-hop, opinion arcs, temporal ordering, budget triage, and numeric aggregation.

Baselines (full OpenAI pipeline):

• v1.0 (9-probe set): 7/9 det · 8/9 judge · 1 halluc — exposed two subject- attribution failures.
• v1.1 (35-probe set): 34/35 det · 35/35 judge · 0 halluc. The single deterministic miss is a check artifact — the correct superseding value ("traded in the Civic for a Tesla") contains the keyword the substring check forbids; the judge scores it correct. This is exactly why the harness grades with both a dumb substring check and an LLM judge.

---

9. Configuration reference

All optional — see .env.example. The service auto-detects providers from whichever keys are present.

Variable	Default	Meaning
DB_PATH	/data/memory.db	SQLite file (on the named volume).
MEMORY_AUTH_TOKEN	(unset)	If set, all endpoints require Authorization: Bearer <token>.
EXTRACTION_PROVIDER	auto	anthropic / openai. Auto-detected from keys; falls back to rule_based.
EXTRACTION_MODEL	per-provider	claude-haiku-4-5 (anthropic) / gpt-4o-mini (openai).
EMBEDDING_PROVIDER	auto	openai / voyage. Auto-detected; falls back to none (BM25-only).
EMBEDDING_MODEL	per-provider	text-embedding-3-small / voyage-3-lite.
ANTHROPIC_API_KEY / OPENAI_API_KEY / VOYAGE_API_KEY	(unset)	Provider credentials.
REQUEST_MAX_BYTES	2097152	Max request body before 413.
LLM_TIMEOUT	45	Per-call HTTP timeout (seconds), within the eval's 60s /turns budget.

---

10. Endpoint reference

Method	Path	Purpose	Success
GET	/health	Liveness/readiness	200 {"status":"ok"}
POST	/turns	Ingest a turn (extract + persist, synchronous)	201 {"id":...}
POST	/recall	Prose context for the next agent turn	200 {context, citations}
POST	/search	Structured search results (tool-call shape)	200 {results[]}
GET	/users/{user_id}/memories	Inspect all memories (incl. superseded)	200 {memories[]}
DELETE	/sessions/{session_id}	Purge a session's data	204
DELETE	/users/{user_id}	Purge a user's data	204

Cross-session scoping (documented design decision): memories are user-scoped and intentionally shared across that user's sessions — that's the point of cross-session memory and fact evolution. Raw-turn recall is likewise scoped to the user. Different user_ids never share. When user_id is null, scope falls back to the session_id so anonymous sessions stay isolated.

---

11. Code map (for maintainers / agents)

Single Python package, src/memory_service/. Each module has a top-of-file docstring explaining its role; this is the 30-second tour:

Module	Responsibility	Start here if…
main.py	FastAPI app: routes, auth dependency, body-size middleware, global error handler, lifespan that wires the store/recall objects.	…you're changing the HTTP surface or status codes.
config.py	Env → Settings. Provider auto-detection, model defaults, enable flags.	…you're adding a provider or a knob.
models.py	Pydantic request/response models. Lenient inputs (extra ignored, content coerced) so odd-but-valid payloads don't 422.	…you're touching the contract shapes.
db.py	SQLite: schema (turns, memories, two FTS5 tables), WAL, lock, read()/write() transaction context managers.	…you're changing storage or indexes.
extraction.py	Turn → candidate memories. LLM prompt + JSON normalization, rule-based regex fallback.	…extraction quality work.
llm.py	Provider-agnostic chat client (Anthropic/OpenAI over httpx) + robust JSON parsing.	…adding an LLM provider.
embeddings.py	Provider-agnostic embeddings + float32 pack/unpack. Returns None when disabled.	…adding an embedding provider.
memory_store.py	Write path: persist turn, reconcile memories (supersede/reinforce), append events, deletes/inspection.	…fact-evolution logic.
recall.py	Read path: query expansion, BM25 + vector ranking, RRF, budget-aware assembly, /search.	…retrieval/ranking work.

Design invariants to preserve when extending:

• /turns stays synchronous — a committed transaction must be immediately recallable. Don't introduce background indexing.
• Every external call (LLM, embeddings) must degrade, not throw, on failure.
• Only active memories are recalled; superseded rows are kept for inspection.
• Scope isolation by user_id (or session_id when null) is a hard boundary — don't widen a query's scope without updating the README's scoping section.
• subject distinguishes the user from third parties; only subject="user" facts belong under "Known facts about this user". Don't collapse the distinction.