Add Comprehensive KnowledgePlane Benchmarking Suite

[altras]

Summary

Comprehensive benchmarking suite + major product improvements for KnowledgePlane.

Key Result: +226% improvement on HotpotQA Supporting Facts F1 vs vector-only retrieval.

---

Core Product Improvements

🔐 Security

• Workspace isolation - Added ownership verification to all REST API endpoints
• Disabled raw AQL - Removed arbitrary query endpoint (IDOR prevention)
• Cross-tenant protection - Normalized workspace ID checks across all /:id routes

🧠 CardConsolidator Enhancements

Change	Impact
Entity + CoT + Confidence extraction	F1: 50% → 57%
Few-shot examples	Better relation quality
Temperature 0.2	More consistent output
Embedding threshold 30% → 45%	Fewer false positives

🎯 BGE Cross-Encoder Reranker (NEW)

• Self-hosted BGE-reranker-v2-m3 as Docker sidecar
• Threshold tuned to 0.40 for optimal precision/recall
• Architecture: Embedding pre-filter → Reranker → LLM verification
• Data sovereignty: No external API leakage (GDPR/HIPAA ready)

✅ LLM Verification for Strong Claims (NEW)

• Verifies causal relations (causes, contradicts, depends_on)
• Based on Zep/Graphiti production pattern
• Result: +6.6pp F1, Precision 45% → 59%

⚡ Embedding Pipeline

• Real-time async processing (5s latency vs 10min sweep)
• Worker triggers for immediate embedding generation
• Rate-limited queue (200 req/min) for API compliance

🔍 Vector Search

• Dynamic nProbe calculation matching nLists
• Full cluster coverage for freshly inserted documents

🤖 Model Upgrade

• gpt-5.1 → gpt-5.2

---

Latest Changes ( by Niki )

⚡ Embedding Triggers Across All Mutation Endpoints

All fact create/update endpoints now queue worker_triggers entries for immediate embedding generation by the background worker (polled every 5s), eliminating the 10-minute sweep delay.

Files changed:

• apps/mcp-server/src/mcp/handlers/facts.write.ts — trigger on single fact create
• apps/mcp-server/src/mcp/handlers/facts.bulkwrite.ts — triggers on bulk fact create
• apps/mcp-server/src/mcp/handlers/facts.update.ts — trigger on content update (skipped if only metadata changed)
• apps/webapp/server/trpc/routes/facts.ts — triggers on tRPC create and update
• docs/SPEC.md — documented trigger-based processing + 10-min sweep as backup

🔍 Vector Index Safety Guards

Hardened vector index creation across all three collections (facts, relations, knowledge_cards):

• Skip index creation when < 16 vectors (FAISS requires training points >= clusters)
• JS cosine fallback handles small collections gracefully
• Simplified nLists: Math.min(vectorCount, 100) — removed Math.max(16, ...) floor that could exceed vector count and crash
• Same guard applied to ensureVectorIndex utility

Files changed:

• packages/db/src/db.ts

🗄️ DB Body Normalization Fix

Replaced Node.js Buffer with standard Uint8Array in normalizeBody for broader runtime compatibility (edge runtimes, Bun, etc.).

Files changed:

• packages/db/src/db.ts

💬 Chat UI Fixes

• Fixed word-break CSS (break-words → wrap-break-word)
• Added explicit text-gray-800 on fact detail boxes for readability in both themes

Files changed:

• apps/webapp/app/chat/page.tsx

🤖 OpenAI SDK Major Upgrade

Upgraded OpenAI SDK from 4.20.0 → ^6.27.0. Removed stale transitive dependencies from lock file (hono, preact, oauth4webapi, etc.).

Files changed:

• packages/aimodel/package.json
• package-lock.json

📦 New Dependencies

Package	Scope	Purpose
p-queue ^9.1.0	background-workers	Controlled concurrency for embedding queue
@next/env ^16.0.4	webapp	Next.js env loading
dotenv-cli ^11.0.0	root (dev)	CLI env management

Files changed:

• apps/background-workers/package.json
• apps/webapp/package.json
• package.json
• package-lock.json

---

Benchmarks

Benchmark	Result	vs Baseline
HotpotQA (Multi-hop)	16.8% SF-F1	+226% vs vector
LongMemEval (Memory)	50% accuracy	92.7% Recall@5
MS-MARCO (Ranking)	0.326 MRR	Competitive
RelationRecall	58% F1	90% recall
Freshness	0.5s	27x faster

LongMemEval Breakdown (ICLR 2025)

• Knowledge Updates: 100% ✅
• Temporal Reasoning: 58%
• Information Extraction: 50%
• Multi-Session Reasoning: 8-17%

Experiments Conducted (20+ runs)

Experiment	Result
Simple 7-rule prompt	50% ✅ Best
Two-Stage LLM	46% (MR +9%, IE -17%)
Aggressive anti-abstention	44% ❌
Chain-of-thought counting	40% ❌

---

Infrastructure

• Docker execution: All benchmarks run in containers
• CLI tool: ./bench hotpot, ./bench longmemeval, ./bench clean
• Experiment tracking: Auto-archives to runs/ with comparison tools
• Reranker sidecar: docker compose --profile with-reranker up

Key Files

tests/benchmarks/
├── bench                    # Main CLI
├── src/
│   ├── hotpotqa.py         # Multi-hop reasoning
│   ├── longmemeval.py      # Memory abilities (ICLR 2025)
│   ├── msmarco.py          # Passage ranking
│   ├── relationrecall.py   # Relation extraction
│   ├── freshness.py        # Write latency
│   └── lib/
│       ├── adapter.py      # KP API client
│       └── preflight.py    # Environment validation
└── docs/
    └── BENCHMARK_EXECUTIVE_SUMMARY.md

apps/background-workers/
├── src/workers/card-consolidator.ts  # Enhanced relation extraction
├── src/services/reranker.py          # BGE cross-encoder

apps/rest-api/
└── src/server.ts                     # Security fixes + trigger-consolidation

Cleanup

• Deleted 26 stale archived docs (-21k lines)
• Consolidated benchmark docs to tests/benchmarks/docs/
• Updated .gitignore for runtime files
• Lock file cleanup: removed unused transitive deps (hono, preact, oauth4webapi, etc.)