AI Memory System - Design Questions

Based on the Strategic Architecture for Hybrid Intelligence report.

1. Existing Projects to Model After

Primary Inspirations (from report):

Project	Role in Your Architecture
Zep	Temporal Knowledge Graphs - facts with validity periods, tracks change over time
Mem0	User personalization, adaptive learning, multi-layer memory, entity tracking
HippoRAG	Associative retrieval via Personalized PageRank (PPR), multi-hop reasoning
Microsoft GraphRAG	Global summarization via community detection (Leiden algorithm)
LangGraph	Orchestration layer - stateful, cyclic agent workflows with routing

Your answer:

Hybrid approach combining multiple systems. Zep for temporal awareness (facts that expire/update), Mem0 for user personalization, HippoRAG-style graph traversal for finding "hidden connections" between facts. The key insight is that relationships between facts are often more important than the facts themselves.

2. Your Unique "Spin"

From the report - the Hybrid Neuro-Symbolic Architecture:

┌─────────────────────────────────────────────────────────────┐
│                    ROUTER (LangGraph)                       │
│         "Is this a vector problem or a graph problem?"      │
└─────────────────┬───────────────────────┬───────────────────┘
                  │                       │
         ┌────────▼────────┐     ┌────────▼────────┐
         │  VECTOR STORE   │     │ KNOWLEDGE GRAPH │
         │  (Semantic)     │     │  (Structural)   │
         │                 │     │                 │
         │ • Fast recall   │     │ • Causal chains │
         │ • Fuzzy match   │     │ • Multi-hop     │
         │ • Unstructured  │     │ • Communities   │
         └────────┬────────┘     └────────┬────────┘
                  │                       │
                  └───────────┬───────────┘
                              │
                    ┌─────────▼─────────┐
                    │   MEMORY LAYER    │
                    │  (Zep + Mem0)     │
                    │                   │
                    │ • User context    │
                    │ • Temporal facts  │
                    │ • Preferences     │
                    └───────────────────┘

Your answer:

Neuro-Symbolic Hybrid: Vectors provide "fast" semantic search (intuition), Graphs provide "slow" causal reasoning (logic). The memory layer adds temporal awareness - facts can expire, be superseded, or evolve. This solves the "multi-hop reasoning failure" where vector search misses connections that require traversing intermediate nodes.

Key differentiator: Facts have validity periods. "Inflation is rising" (Q1) can be superseded by "Inflation is falling" (Q2). Standard vector retrieval returns both; temporal graphs know Q2 supersedes Q1.

3. Target Projects & Tech Stack

Primary Target: daily-advisor-panel

Aspect	Decision
Use Case	AI advisors that help users with their day, emails, calendar, and business problems
Memory Needs	User preferences, strategic priorities, past decisions, recurring patterns
Example Query	"Should we delay the launch given supply chain issues?"
Required Reasoning	Multi-hop (connect disruption → component → product → revenue target)

Your answer:

First target is daily-advisor-panel. The advisors need to:

Remember user's strategic priorities ("Q4 revenue is #1 priority")

Recall relevant past decisions and their outcomes

Connect disparate facts (calendar + emails + business context)

Provide personalized advice colored by user preferences

Tech stack: Python backend (matches auto-twitter-stoic), can expose via REST API for other projects.

4. Memory Types & Architecture

All four types are essential for strategic advising:

Short-term / Working Memory

Current conversation thread
Immediate task context
Implementation: LangGraph state object
Lifespan: Session

Episodic Memory

"As we discussed in the board meeting last Tuesday..."
Specific events with timestamps
Implementation: Zep temporal graph
Lifespan: Months (with decay)

Semantic Memory

"User prioritizes EBITDA over revenue growth"
"Project Alpha is strictly confidential"
Facts extracted from interactions
Implementation: Knowledge graph nodes + Mem0 user profiles
Lifespan: Long-term, updated incrementally

Procedural Memory

"User prefers concise answers without excessive caveats"
"User always wants risks highlighted first"
Learned patterns from behavior
Implementation: Mem0 adaptive personalization
Lifespan: Long-term, refined over time

Your answer:

All four memory types are essential for strategic advising. The report emphasizes that storing everything in context window causes "Lost in the Middle" and explodes token costs. Need dedicated memory systems that store, index, and retrieve based on relevance.

5. Technical Constraints

Architecture from report:

Component	Technology	Purpose
Orchestration	LangGraph	Stateful agent workflows, routing, loops
Integration	MCP (Model Context Protocol)	Standardized tool interface for LLMs
Vector Store	Pinecone/Chroma/pgvector	Semantic similarity search
Graph Store	Neo4j or embedded	Entity relationships, traversal
Memory Service	Zep or Mem0	Temporal facts, user personalization

Key patterns:

Router Pattern: Analyze query → route to vector/graph/hybrid
Reciprocal Rank Fusion (RRF): Combine vector + graph results
MCP Servers: Wrap each data source as standardized tool

Your answer:

Orchestration: LangGraph (enables self-correction loops, not just linear chains)

Integration: MCP for "Strategic Data Fabric" - decouple agent logic from data infrastructure

Latency: Accept that strategic reasoning takes time; use streaming to show "thinking"

Deployment: Self-hosted initially, can scale to cloud

6. Standalone vs Integrated - Decision Framework

Build Standalone If:

You want to use it in 3+ projects (daily-advisor, audio-notetaker, stoic-ghostwriter, auto-twitter)
Projects use different tech stacks (React Native, Python, n8n)
You want to version/deploy memory system independently
Other developers might use it
You want a clean separation of concerns

Hybrid Approach (RECOMMENDED):

Build as a standalone service but start by integrating into daily-advisor-panel as the proving ground. The report explicitly recommends this: "MCP creates a 'Strategic Data Fabric' - instead of hard-coding database connections, your agent requests 'tools' from the MCP registry."

Your decision:

Standalone service with MCP interface, first integrated into daily-advisor-panel. This allows:

Other projects to connect via standardized MCP protocol

Swapping storage backends without rewriting agent logic

Independent versioning and deployment

Summary of Decisions

Decision	Choice
Primary inspiration	Zep (temporal) + Mem0 (personalization) + HippoRAG (multi-hop)
Unique differentiator	Hybrid neuro-symbolic with temporal fact validity
First target project	daily-advisor-panel
Required tech stack	Python, LangGraph, MCP, Neo4j/pgvector
Memory types (MVP)	All four: short-term, episodic, semantic, procedural
Storage approach	Hybrid: Vector + Knowledge Graph + Temporal Memory
Standalone vs integrated	Standalone service, first integrated into daily-advisor
Deployment model	Self-hosted, MCP interface

Architecture Implications for ai-memory-system

Based on these decisions, the skeleton needs expansion:

src/ai_memory/
├── core/                 # ✅ Done
│   ├── types.py          # Need to add temporal validity fields
│   └── memory.py         # Need Router pattern
├── storage/
│   ├── sqlite.py         # ✅ Done (dev/testing)
│   ├── vector.py         # NEW: Vector store backend
│   └── graph.py          # NEW: Knowledge graph backend
├── retrieval/
│   ├── base.py           # ✅ Done
│   ├── vector.py         # NEW: Semantic similarity
│   ├── graph.py          # NEW: PPR traversal (HippoRAG-style)
│   └── hybrid.py         # NEW: RRF fusion
├── temporal/             # NEW: Zep-style temporal layer
│   ├── facts.py          # Facts with validity periods
│   └── supersession.py   # Fact update/expiry logic
├── personalization/      # NEW: Mem0-style user layer
│   ├── profiles.py       # User preference tracking
│   └── patterns.py       # Behavioral pattern detection
├── orchestration/        # NEW: LangGraph integration
│   ├── router.py         # Query classification + routing
│   └── workflows.py      # Agent workflow definitions
└── api/
    ├── rest.py           # REST API (FastAPI)
    └── mcp.py            # NEW: MCP server implementation

Next Steps

Extend types.py - Add temporal validity (valid_from, valid_until, superseded_by)
Implement Router - Query classifier that routes to vector/graph/hybrid
Add graph storage - Neo4j or embedded graph for relationship traversal
Build MCP server - Standardized interface for LLM tool use
Integrate with daily-advisor-panel - First real-world test

Troubleshooting & Edge Cases

Detailed decisions for handling edge cases in the memory system.

1. Cold Start

Problem: New user has empty graph, system has nothing to retrieve.

Decision: Integration bootstrap

First session seeds basic structure
Graceful handling of empty results
Progressive enrichment as conversations accumulate

2. Entity Disambiguation

Problem: "Sarah" could be multiple people.

Decision: Embedding + user clarification

Use embedding similarity to match likely entities
When ambiguous, ask user: "Do you mean Sarah (CFO) or Sarah (vendor)?"
Store disambiguation choices for future reference

3. Fact Conflicts

Problem: "Meeting is Monday" then later "Meeting moved to Friday"

Decision: Hybrid detection + supersession

Detect conflicting facts via semantic similarity
Later facts supersede earlier ones (temporal validity)
Keep superseded facts with valid_until timestamp for audit

4. Neo4j Failure Handling

Problem: Graph database unavailable.

Decision: Graceful degradation with queue

Queue writes when Neo4j is down
Return partial results (vector-only if graph unavailable)
Replay queued operations when connection restored

5. Extraction Accuracy

Problem: LLM may miss entities or extract incorrectly.

Decision: Tiered signal detection with hybrid extraction model

Extraction Model

Signal Level	Action
HIGH (entities, dates, preferences)	Extract immediately
MEDIUM (ambiguous)	Queue for batch processing
LOW (greetings, filler)	Ignore

Signal Detection (Tiered)

Message → Regex patterns (0ms)
              │
              ├─ HIGH match → extract now
              ├─ LOW match → ignore
              └─ UNCERTAIN → Haiku classifier (~200ms)

Regex Pattern Categories

Entities: Named people, roles, companies
Dates: Deadlines, timeframes (Q1, next week, Friday)
Preferences: "I prefer/hate/always/never"
Decisions: "decided/agreed/approved"
Priorities: "top priority/critical/urgent"

Queue Storage

SQLite table (extraction_queue)
Persists across restarts
Easy debugging/inspection

Batch Triggers (Hybrid)

Trigger	Condition
Explicit close	`memory.end_session()`
Inactivity	15 minutes no messages
Queue size	50 items accumulated
Shutdown	Process exit / SIGTERM

6. Performance Bottlenecks

Problem: Memory operations add latency to every conversation.

Decisions:

Aspect	Decision	Rationale
Async extraction	Yes (BackgroundTasks)	Don't block user responses
Retrieval caching	Session-scoped	Fast within conversation, fresh per session
Preloading	Background on session start	Load likely-needed entities before first query
Parallel retrieval	Adaptive (router-based)	Vector-only for simple queries, both for complex

Voice-First Considerations

daily-advisor-panel is voice-first
ElevenLabs STT is non-streaming (~500-1000ms)
Natural conversation pace absorbs memory latency
200ms added latency is acceptable

Performance Profile

Scenario	Added Latency
Simple query (vector only)	~50ms
Complex query (graph + vector)	~300ms
Extraction	0ms (background)

7. Privacy & Sensitive Data

Problem: Users discuss sensitive business information.

Decisions:

Aspect	Decision	Rationale
Multi-tenant	Skip for MVP	Single user system initially
Storage	Self-hosted (S2)	Data stays on your server
LLM exposure	Current message only	Full graph never sent to cloud
PII filtering	Basic regex	Filter passwords, API keys, SSNs before extraction
Deletion	Basic wipe	"Delete all" for MVP, selective later

Self-Hosted Architecture (S2)

┌─────────────────────────────────────────┐
│              YOUR SERVER                 │
│  ┌─────────────┐  ┌─────────────────┐   │
│  │   Neo4j     │  │     Chroma      │   │
│  │  (graph)    │  │   (vectors)     │   │
│  └──────┬──────┘  └────────┬────────┘   │
│         └────────┬─────────┘            │
│           ┌──────▼──────┐               │
│           │  ai-memory  │               │
│           └──────┬──────┘               │
└──────────────────┼──────────────────────┘
                   │ (API calls only)
                   ▼
          Cloud LLM (Claude/OpenAI)

Why S2 over S4 (SQLite)?

Neo4j enables "impressive" UX: hidden connection discovery, causal chains, proactive insights
Multi-hop queries are native and fast
Worth the infrastructure for "wow" moments

8. Testing Strategy

Problem: Memory system is stateful, involves LLMs, has graph traversal.

Decisions:

Aspect	Decision
LLM extraction	Eval framework (quality scoring)
Test data	Fixtures + Factories
Neo4j in tests	Testcontainers (real DB, isolated)
Coverage focus	80% unit, 20% integration/E2E

Test Structure

tests/
├── unit/
│   ├── test_signal_detection.py    # Regex patterns
│   ├── test_entity_parsing.py      # Entity extraction
│   ├── test_temporal_logic.py      # Supersession, expiry
│   └── test_rrf_fusion.py          # Result combination
├── integration/
│   ├── test_neo4j_storage.py       # CRUD operations
│   ├── test_graph_traversal.py     # PPR, path queries
│   └── test_embedding_search.py    # Vector similarity
├── e2e/
│   └── test_memory_flow.py         # Full conversation scenarios
├── evals/
│   ├── extraction_quality.py       # LLM extraction accuracy
│   └── retrieval_relevance.py      # Retrieved context quality
└── fixtures/
    ├── sample_graph.json           # Pre-built test graph
    └── conversations.json          # Test scenarios

9. API Contracts

Problem: How do applications integrate with the memory system?

Decisions:

Aspect	Decision
Interface types	Library + REST + MCP
Sync/Async	Async primary, sync wrapper
Format	Structured with sensible defaults

Core Operations

Operation	Purpose
`store_message`	Save user/assistant exchange
`extract_entities`	Parse entities from text
`query_memory`	Retrieve specific facts
`get_context`	Get relevant context for query
`delete_all`	Wipe user data

Integration Example

from ai_memory import MemoryStore

memory = MemoryStore(neo4j_uri, user_id)

# Before generating response
context = await memory.get_context(user_message)

# After response (background)
await memory.save_exchange(user_message, assistant_response)

10. Rollback & Recovery

Problem: What happens when things go wrong?

Decisions:

Scenario	Solution
Bad extraction	Soft delete (mark invalid, keep audit trail)
Database backup	Scheduled daily dumps
Deployment	Schema versioning with migrations

Soft Delete Pattern

class Entity:
    id: str
    name: str
    valid: bool = True
    invalidated_at: datetime | None = None
    invalidated_reason: str | None = None

Schema Versioning

SCHEMA_VERSION = 2

def migrate(current_version):
    if current_version < 2:
        run_migration_v2()

Summary of All Decisions

Core Architecture

Decision	Choice
Graph storage	Neo4j (self-hosted)
Vector storage	Chroma (self-hosted)
Embeddings	Local (sentence-transformers)
LLM	Cloud (OpenAI/Claude)
Target UX	Impressive ("wow" moments with hidden connections)

Extraction & Performance

Decision	Choice
Signal detection	Tiered (regex → Haiku fallback)
Extraction timing	Async (BackgroundTasks)
Queue storage	SQLite
Batch triggers	Hybrid (explicit, timeout, threshold, shutdown)
Caching	Session-scoped
Retrieval	Adaptive (router decides vector/graph/both)

Operations

Decision	Choice
API interface	Library + REST + MCP
Testing	Eval framework, testcontainers, fixtures
Backup	Daily scheduled
Rollback	Soft delete + schema versioning
Privacy	Self-hosted storage, basic PII filter

FilesExpand file tree

DESIGN-QUESTIONS.md

Latest commit

History

DESIGN-QUESTIONS.md

File metadata and controls

AI Memory System - Design Questions

1. Existing Projects to Model After

2. Your Unique "Spin"

3. Target Projects & Tech Stack

4. Memory Types & Architecture

Short-term / Working Memory

Episodic Memory

Semantic Memory

Procedural Memory

5. Technical Constraints

6. Standalone vs Integrated - Decision Framework

Build Standalone If:

Hybrid Approach (RECOMMENDED):

Summary of Decisions

Architecture Implications for ai-memory-system

Next Steps

Troubleshooting & Edge Cases

1. Cold Start

2. Entity Disambiguation

3. Fact Conflicts

4. Neo4j Failure Handling

5. Extraction Accuracy

Extraction Model

Signal Detection (Tiered)

Regex Pattern Categories

Queue Storage

Batch Triggers (Hybrid)

6. Performance Bottlenecks

Voice-First Considerations

Performance Profile

7. Privacy & Sensitive Data

Self-Hosted Architecture (S2)

8. Testing Strategy

Test Structure

9. API Contracts

Core Operations

Integration Example

10. Rollback & Recovery

Soft Delete Pattern

Schema Versioning

Summary of All Decisions

Core Architecture

Extraction & Performance

Operations