Skip to content

Commit e2c28b8

Browse files
uogbujiuogbuji
committed
Implement unified knowldgebase backend
1 parent eb0394b commit e2c28b8

9 files changed

Lines changed: 1529 additions & 6 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
PLAN.md
2+
13
**/.env
24
env.sh
35
.DS_Store

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,17 @@ Notable changes to Format based on [Keep a Changelog](https://keepachangelog.co
55
## [Unreleased]
66

77
### Added
8+
- **UnifiedKB API**: Single interface for managing multiple knowledge base backends
9+
- `UnifiedKB` class in `ogbujipt.memory.unified` orchestrates multiple backends
10+
- Backend registration with `add_backend()`, `remove_backend()`, and metadata tracking
11+
- Backend management: `enable_backend()`, `disable_backend()`, `list_backends()`
12+
- Parallel search execution across backends with automatic result aggregation
13+
- Parallel `insert()` and `delete()` operations to multiple backends
14+
- Weight-based scoring to prioritize certain backends
15+
- Error isolation - one backend failure doesn't break entire operation
16+
- Selective operations - target all backends or specific ones
17+
- Comprehensive test suite (27 unit tests) in `test/memory/test_unified.py`
18+
- Demo and documentation in `demo/unified-kb/`
819
- **Onya Knowledge Graph Support**: New `OnyaKB` backend for loading and searching `.onya` files from directories
920
- File-based knowledge graph storage without database overhead
1021
- Compatible with `KBBackend` protocol for unified KB system integration

‎README.md‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,37 @@ async for result in hybrid.execute('machine learning', backends=[kb], limit=5):
8181
print(f'{result.score:.3f}: {result.content[:50]}...')
8282
```
8383

84+
### Unified knowledge base API
85+
86+
Manage multiple backends (RAM, PostgreSQL, Qdrant, Onya graphs) with a single interface:
87+
88+
```py
89+
from ogbujipt.memory.unified import UnifiedKB
90+
from ogbujipt.store.ram import RAMDataDB
91+
from ogbujipt.store.postgres import DataDB
92+
93+
# Create backends
94+
ram_kb = RAMDataDB(embedding_model=model, collection_name='cache')
95+
await ram_kb.setup()
96+
97+
pg_kb = await DataDB.from_conn_params(
98+
host='localhost', db_name='knowledge',
99+
embedding_model=model, table_name='documents'
100+
)
101+
102+
# Create unified KB and register backends
103+
kb = UnifiedKB()
104+
kb.add_backend('cache', ram_kb, weight=1.0)
105+
kb.add_backend('persistent', pg_kb, weight=1.5)
106+
107+
# Search across all backends automatically
108+
async for result in kb.search('machine learning', limit=10):
109+
print(f'{result.score:.3f} [{result.source}]: {result.content[:50]}')
110+
111+
# Insert to all backends
112+
await kb.insert('New content', metadata={'topic': 'AI'})
113+
```
114+
84115
## Knowledge bank features
85116

86117
OgbujiPT provides a flexible knowledge bank system with multiple storage backends and retrieval strategies.
@@ -204,13 +235,17 @@ See the [`demo/`](https://github.com/OoriData/OgbujiPT/tree/main/demo) directory
204235

205236
### Knowledge bank demos
206237

238+
- **`unified-kb/`**: Unified knowledge base API
239+
- `simple_unified_demo.py`: Managing multiple backends with automatic aggregation
207240
- **`ram-store/`**: In-memory vector stores—zero setup, perfect for learning
208241
- `simple_search_demo.py`: Basic semantic search with filtering
209242
- `chat_with_memory.py`: Conversational AI with message history
210243
- **`pg-hybrid/`**: PostgreSQL-based production examples
211244
- `chat_with_hybrid_kb.py`: Hybrid search with RRF fusion
212245
- `hybrid_rerank_demo.py`: Reranking with cross-encoders
213246
- `chat_doc_folder_pg.py`: RAG chat application
247+
- **`kgraph/`**: Onya knowledge graph integration
248+
- `simple_onya_demo.py`: Graph-based knowledge retrieval
214249

215250
### LLM demos
216251

@@ -226,20 +261,21 @@ OgbujiPT is evolving into a comprehensive knowledge bank system. Current focus (
226261

227262
### ✅ Implemented
228263

264+
- **Unified KB API**: Single interface for multiple backends with automatic aggregation
229265
- In-memory vector stores (RAMDataDB, RAMMessageDB)
230266
- Dense vector search (PostgreSQL, Qdrant, in-memory)
231267
- Sparse retrieval (BM25)
232268
- Hybrid search with RRF fusion
233269
- Cross-encoder reranking
270+
- GraphRAG support using [Onya](https://github.com/OoriData/Onya)
234271
- Message/conversation storage
235272
- Metadata filtering
236273

237274
### 🚧 In progress
238275

239-
- GraphRAG support using [Onya](https://github.com/OoriData/Onya)
240-
- Unified knowledge base API
241-
- Query classification and routing
242-
- Multi-backend aggregation
276+
- Query classification and intelligent routing
277+
- Score normalization across backend types
278+
- Result deduplication
243279

244280
### 📋 Planned
245281

‎demo/unified-kb/README.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
**UnifiedKB Demos**. Demonstrations of the `UnifiedKB` API—a unified interface for managing multiple knowledge base backends with automatic result aggregation, supporting:
2+
3+
- **In-memory stores** (RAMDataDB, RAMMessageDB)
4+
- **PostgreSQL + pgvector** (DataDB, MessageDB)
5+
- **Qdrant** vector database
6+
- **Onya** knowledge graphs
7+
- Any backend implementing the `KBBackend` protocol
8+
9+
# Key Features
10+
11+
- **Unified interface**: One API for all backends
12+
- **Automatic aggregation**: Results from multiple backends automatically merged and sorted
13+
- **Parallel execution**: Searches run concurrently across backends using asyncio
14+
- **Backend management**: Enable/disable backends without removing them
15+
- **Selective operations**: Target all backends or specific ones
16+
- **Weight-based scoring**: Prioritize results from certain backends
17+
- **Error isolation**: One backend failure doesn't break the entire operation
18+
19+
# Demos
20+
21+
## `simple_unified_demo.py`
22+
23+
Complete introduction to UnifiedKB showing:
24+
25+
1. Setting up multiple backend stores
26+
2. Registering backends with UnifiedKB
27+
3. Inserting documents across all backends
28+
4. Searching and aggregating results
29+
5. Searching specific backends only
30+
6. Managing backends (enable/disable)
31+
7. Selective insertion to specific backends
32+
33+
**Requirements:**
34+
```bash
35+
uv pip install ogbujipt sentence-transformers
36+
```
37+
38+
**Run:**
39+
```bash
40+
python demo/unified-kb/simple_unified_demo.py
41+
```
42+
43+
**Expected output:**
44+
```
45+
======================================================================
46+
UnifiedKB Demo: Unified Knowledge Base API
47+
======================================================================
48+
49+
[1] Loading embedding model...
50+
✓ Model loaded: all-MiniLM-L6-v2
51+
52+
[2] Setting up backend stores...
53+
✓ Backend 1: General programming KB (in-memory)
54+
✓ Backend 2: Language-specific KB (in-memory)
55+
✓ Backend 3: Cache KB (in-memory)
56+
57+
[3] Creating UnifiedKB and registering backends...
58+
✓ Registered 3 backends
59+
60+
[4] Backend information:
61+
• general:
62+
- Weight: 1.5
63+
- Enabled: True
64+
- Scope: general
65+
...
66+
```
67+
68+
# Usage Patterns
69+
70+
## Basic Setup
71+
72+
```python
73+
from ogbujipt.memory.unified import UnifiedKB
74+
from ogbujipt.store.ram import RAMDataDB
75+
from sentence_transformers import SentenceTransformer
76+
77+
# Create backends
78+
model = SentenceTransformer('all-MiniLM-L6-v2')
79+
backend1 = RAMDataDB(embedding_model=model, collection_name='kb1')
80+
await backend1.setup()
81+
82+
# Create unified KB
83+
kb = UnifiedKB()
84+
kb.add_backend('main', backend1, weight=1.5,
85+
metadata={'type': 'vector', 'scope': 'general'})
86+
```
87+
88+
## Searching All Backends
89+
90+
```python
91+
# Searches all enabled backends in parallel
92+
async for result in kb.search('machine learning', limit=10):
93+
print(f'{result.score:.3f} [{result.source}]: {result.content}')
94+
```
95+
96+
## Searching Specific Backends
97+
98+
```python
99+
# Search only certain backends
100+
async for result in kb.search('query', backends=['main', 'cache']):
101+
print(result.content)
102+
```
103+
104+
## Inserting Content
105+
106+
```python
107+
# Insert to all enabled backends (default)
108+
result_ids = await kb.insert(
109+
'Python is great for ML',
110+
metadata={'topic': 'programming'}
111+
)
112+
113+
# Insert to specific backends only
114+
result_ids = await kb.insert(
115+
'Temporary data',
116+
backends=['cache'],
117+
metadata={'ttl': 3600}
118+
)
119+
```
120+
121+
## Managing Backends
122+
123+
```python
124+
# List all backends
125+
for name, info in kb.list_backends().items():
126+
print(f'{name}: weight={info["weight"]}, enabled={info["enabled"]}')
127+
128+
# Temporarily disable a backend
129+
kb.disable_backend('slow_backend')
130+
131+
# Re-enable it later
132+
kb.enable_backend('slow_backend')
133+
134+
# Remove backend completely
135+
kb.remove_backend('old_backend')
136+
```
137+
138+
## Backend Weights
139+
140+
Weights influence result scoring during aggregation:
141+
142+
```python
143+
# Higher weight = results from this backend ranked higher
144+
kb.add_backend('primary', pg_store, weight=2.0) # Most important
145+
kb.add_backend('secondary', ram_store, weight=1.0) # Standard
146+
kb.add_backend('cache', cache_store, weight=0.5) # Less important
147+
```
148+
149+
# Design Philosophy
150+
151+
## Composability over Monolith
152+
153+
UnifiedKB is a thin orchestration layer that doesn't duplicate backend capabilities—it coordinates them. Each backend maintains its own:
154+
- Connection pools
155+
- Embedding models
156+
- Indexing strategies
157+
- Configuration
158+
159+
## Explicit over Implicit
160+
161+
No hidden magic:
162+
- Parallel execution is explicit (using asyncio.gather())
163+
- Error handling is clear (one backend failure doesn't break others)
164+
- Backend selection is explicit (via `backends` parameter)
165+
- Scoring/aggregation is transparent (simple sort by score)
166+
167+
## Protocol-based Design
168+
169+
Any object implementing the `KBBackend` protocol works:
170+
171+
```python
172+
class MyCustomBackend:
173+
async def search(self, query, limit=5, **kwargs):
174+
# Your implementation
175+
...
176+
177+
async def insert(self, content, metadata=None, **kwargs):
178+
# Your implementation
179+
...
180+
181+
async def delete(self, item_id, **kwargs):
182+
# Your implementation
183+
...
184+
```
185+
186+
# Advanced Topics
187+
188+
## Result Aggregation
189+
190+
Currently, UnifiedKB uses simple score-based aggregation:
191+
1. Execute searches in parallel across all selected backends
192+
2. Collect all results
193+
3. Sort by score (descending)
194+
4. Return top N results
195+
196+
**Coming soon:**
197+
- Score normalization across different backend types
198+
- Result deduplication (fuzzy matching)
199+
- Cross-encoder re-ranking
200+
- Query classification and intelligent routing
201+
202+
## Error Handling
203+
204+
Backend operations are isolated:
205+
- Search: Failed backends are logged but don't block others
206+
- Insert: Partial success is possible (some backends succeed, others fail)
207+
- Delete: Each backend's result tracked independently
208+
209+
## Performance Considerations
210+
211+
- **Parallel execution**: All backend operations run concurrently
212+
- **Streaming results**: Results yielded as they're aggregated (not all buffered)
213+
- **Timeout support**: Can be added per-backend if needed
214+
- **Connection pooling**: Handled by each backend independently
215+
216+
# Next Steps
217+
218+
After exploring this demo, check out:
219+
- `demo/ram-store/` - In-memory backend examples
220+
- `demo/pg-hybrid/` - PostgreSQL hybrid search examples
221+
- `demo/kgraph/` - Onya knowledge graph examples
222+
- `test/memory/test_unified.py` - Comprehensive unit tests
223+
224+
# Roadmap
225+
226+
Upcoming UnifiedKB features:
227+
- Query classification for intelligent routing
228+
- Score normalization strategies
229+
- Result deduplication (fuzzy matching)
230+
- Integration with search strategies (hybrid, sparse, dense)
231+
- MCP (Model Context Protocol) support
232+
- Advanced aggregation (RRF, cross-encoder reranking)

0 commit comments

Comments
 (0)