Skip to content

plan.md

Latest commit

 

History

History
639 lines (523 loc) · 26.1 KB

plan.md

File metadata and controls

639 lines (523 loc) · 26.1 KB

Agentic Search Platform - Project Plan

🎯 Project Overview

Build a next-generation intelligent agentic search platform that beats traditional RAG by 3-5x in speed and 60-70% in cost through:

  • Adaptive Compression: Content-aware OCR with DeepSeek Vision (10x+ compression)
  • Speculative Execution: Prefetch documents and start processing before queries complete
  • Hybrid Storage: LanceDB vectors + knowledge graphs + BM25 keyword search
  • Real-Time Streaming: Progressive results with parallel segment execution
  • Multi-Modal OCR: Process images, tables, charts, and diagrams
  • Continuous Learning: Human-in-the-loop feedback for fine-tuning

Deployed to Cloudflare at mikepfunk.com with multi-model support (local + cloud).

🏗️ Infrastructure Setup

Phase 1: Deployment & Configuration (Priority: CRITICAL)

  • Fix Cloudflare build error (.output/server directory missing)
  • Create wrangler.json for Cloudflare Pages deployment
  • Configure build output for TanStack Start + Cloudflare
  • Setup environment variables in Cloudflare dashboard
  • CI/CD GitHub Actions fixed (pnpm, master branch, wrangler-action)
  • Dependabot weekly dependency updates configured
  • Test successful deployment to mikepfunk.com
  • Configure custom domain DNS (mikepfunk.com → Cloudflare Pages)

Phase 2: Backend Services

  • Convex Backend

    • Run npx convex dev and initialize project
    • Create schema for model configs, chat history, search results (20+ tables)
    • Setup real-time subscriptions for chat
    • Configure Convex authentication (GitHub OAuth + Password + Anonymous)

  • Sentry Integration (mikepfunk.sentry.io)

    • Sentry already installed (@sentry/react, @sentry/tanstackstart-react)
    • Configure DSN in environment variables (VITE_SENTRY_DSN set)
    • Add performance monitoring for API routes (TanStack Start integration)
    • Setup error boundaries for React components
    • Add breadcrumbs for user actions

  • CodeRabbit CI/CD

    • Add .coderabbit.yaml configuration
    • Setup GitHub Actions workflow
    • Configure PR review automation
    • Add code quality checks

🤖 Core Features

Phase 3: Model Selection & Integration

  • Model configuration UI (Settings page)
  • Support for 6 providers (OpenAI, Anthropic, Google, Ollama, LM Studio, Azure)
  • Web Crypto API encryption for API keys
  • CSRF protection for API routes
  • Convex Schema for Models
    • modelConfigurations table
    • mcpServers table
    • User preferences table
  • Local Model Integration (Ollama)
    • Auto-detect Ollama running on localhost:11434
    • List available Ollama models via API
    • Test connection without API key
    • Fallback to cloud models if local unavailable
  • MCP Server Integration
    • Connect model selection to MCP servers
    • Create MCP configuration UI
    • Test with claude-flow MCP server
    • Support custom MCP servers

Phase 4: Agentic Search with Chat Interface

  • Chat UI Component

    • Build ChatInterface.tsx with message history (AgenticChat component)
    • Add SearchBar integration
    • Stream responses from AI models (SSE streaming)
    • Display search results inline
    • Markdown rendering with syntax highlighting

  • Search Backend

    • Create /api/search endpoint (stream.ts with SSE)
    • Integrate with selected AI model (local or cloud via unified-provider)
    • Parse user intent from chat message
    • Execute multi-step agentic search (UnifiedSearchOrchestrator)
    • Return structured results (sources, summaries, links)

  • Agentic Search Logic

    • Break down complex queries into sub-queries (segment execution)
    • Parallel search across multiple providers
    • Aggregate results from multiple sources
    • Rank and deduplicate results (ADD discriminator scoring)
    • Provide source attribution

Phase 5: Memory Management (Short-term + Long-term)

  • Short-term Memory (Convex)

    • Store chat history per session (Convex real-time)
    • Cache recent search results (5-minute TTL via semantic cache)
    • User context and preferences
    • Active model selection state

  • Long-term Memory (S3 + Persistence)

    • S3: Store large search result datasets (s3-storage.ts with AES256)
    • PersistenceAdapter interface for pluggable backends
    • Finetuning dataset export to S3 (exportToS3)
    • Archive old chat sessions (> 30 days)
    • Full-text search across historical data
    • User analytics and usage patterns (searchAnalytics)

  • Memory Retrieval

    • Semantic search across conversation history (semantic cache)
    • Context injection for follow-up queries (query enhancement pipeline)
    • Personalized recommendations based on history
    • Privacy controls (delete history, export data)

🚀 Advanced Features (Beyond RAG)

Phase 6: Multi-Modal OCR with DeepSeek Vision

  • DeepSeek Vision Integration

    • Add vision OCR module (src/lib/ocr/deepseek-vision.ts)
    • Process images via multimodal AI models (6 vision providers)
    • Layout-aware extraction (preserve structure as markdown)
    • Multimodal understanding (images + text together)
    • Progressive OCR streaming

  • Adaptive Compression Strategy

    • Content-aware compression ratios:
      • Legal/medical: 3-5x (high detail preservation)
      • News/blogs: 10-15x (aggressive compression)
      • Code: 2-3x (preserve syntax)
      • Technical docs: 5-8x (balanced)
    • Query-aware decompression (expand relevant sections)
    • Hierarchical compression (paragraph/section/document)
    • Compression confidence scoring

Phase 7: Hybrid Vector + Graph Storage

  • LanceDB Integration

    • Setup LanceDB for fast vector search
    • Hybrid search: vectors + SQL capabilities
    • Store embeddings with metadata
    • Create indexes for common query patterns
    • In-memory fallback with cosine similarity

  • Knowledge Graph

    • Entity extraction and relationship mapping
    • Semantic connections between documents
    • Graph-based query expansion
    • Relationship-aware retrieval
    • Serialization for persistence

  • Multi-Index Strategy

    • BM25 for keyword search
    • Vector embeddings for semantic search
    • Graph traversal for relationship queries
    • Hybrid ranking algorithm
    • Query routing based on type

Phase 8: Speculative Execution & Prefetching

  • Query Intent Prediction

    • Start segmentation before user finishes typing
    • Predict likely follow-up queries
    • Preload related documents
    • Cache predicted results

  • Parallel Document Pre-fetching

    • Fetch likely documents during reasoning
    • Background indexing during idle time
    • Smart prefetch based on user patterns
    • Priority queue for hot documents

  • Result Caching with Prediction

    • Semantic caching (similar queries)
    • Cache likely follow-up queries
    • Partial result caching (segment-level)
    • Smart cache invalidation

Phase 9: Real-Time Streaming Architecture

  • Progressive Enhancement

    • Stream results as they arrive
    • Display partial/incomplete information immediately
    • Enhance quality progressively
    • User interruption support (stop/redirect)

  • Stream-First Pipeline

    • Parallel segment execution with streaming
    • Live token usage and confidence metrics
    • Real-time reasoning step visualization
    • Incremental result aggregation

Phase 10: Advanced Caching Strategies

  • Semantic Caching

    • Vector similarity matching for queries
    • Match similar queries, not just exact
    • Confidence-based cache hits
    • Query normalization and canonicalization

  • Multi-Tier Caching

    • Memory (hot cache, <1ms)
    • Redis (warm cache, <10ms)
    • LanceDB (vector cache, <100ms)
    • S3 (cold storage, <1s)
    • Smart tier promotion/demotion

  • Incremental Indexing

    • Delta updates (only changed sections)
    • Document versioning with diffs
    • Smart cache invalidation (affected entries only)
    • Preemptive indexing (before queries)

Phase 11: Query Enhancement Pipeline

  • Query Rewriting

    • Spelling correction (typo fixing)
    • Entity recognition and normalization
    • Query expansion (synonyms, related terms)
    • Context injection (user history)
    • Multi-language support (translation) - 10 languages supported

  • Confidence-Based Model Routing

    • Dynamic routing per segment type (model-routing module)
    • Query complexity classification
    • Cost-aware routing decisions
    • Manual override support + local model preference
    • Fallback chain generation

Phase 12: LangSmith & OpenTelemetry Observability

  • Observability Service
    • Distributed tracing across search operations
    • Span attributes for all operations
    • Custom metrics (cache hit rate, latency, tokens)
    • Search trace recording
    • Model call trace recording
    • LangSmith integration (API key setup pending)
    • Performance monitoring dashboards
    • Alerting on degraded performance

🔧 Technical Integrations

Phase 13: Claude Flow MCP Integration

  • Claude Flow initialized with mesh swarm

  • ReasoningBank memory enabled

  • MCP Tools for Search

    • Use mcp__claude-flow__task_orchestrate for complex searches
    • Spawn researcher agents for deep dives
    • Use memory system for context persistence
    • Integrate with local models (Ollama via MCP)

  • Custom MCP Servers

    • Create search-specific MCP tools
    • Web scraping MCP server
    • Document parsing MCP server
    • Knowledge graph MCP server

Phase 7b: AI SDK Provider Adapter

  • Create unified provider interface (src/lib/ai/unified-provider.ts)
  • Map ModelConfigManager → AI SDK providers (13 providers supported)
  • SSRF validation on all provider base URLs
  • Handle provider-specific features (tools, vision, etc.)
  • Automatic fallback on provider errors

🧪 Testing & Quality Assurance

Phase 8: Comprehensive Testing

  • 244+ tests passing across 10+ test files

  • 29 tests passing for CSRF protection

  • E2E Tests (Playwright)

    • Search flow tests (6 tests)
    • Settings flow tests (4 tests)
    • History flow tests (6 tests)
    • Test chat interface with streaming
    • Test memory persistence

  • Unit Tests

    • ADD discriminator (13 tests)
    • URL validation (16 tests)
    • Unified provider (10 tests)
    • DeepSeek vision OCR (8 tests)
    • Persistence adapter (14 tests)
    • Translation service (35+ tests)
    • Model routing (7 tests)
    • Knowledge graph (5 tests)
    • Vector storage (5 tests)

  • Integration Tests

    • Convex real-time sync
    • S3/DynamoDB operations
    • MCP server connectivity
    • Cloudflare deployment

  • Performance Tests

    • Search latency benchmarks
    • Memory usage profiling
    • Concurrent user load testing
    • Bundle size optimization

🚀 Deployment Pipeline

Phase 9: CI/CD with Cloudflare

  • GitHub Actions Workflow

    • Build and test on PR (pnpm + vitest + typecheck)
    • Deploy via wrangler-action on push to master
    • Deploy preview for each PR (Cloudflare preview env)
    • CodeRabbit automated reviews
    • Add Dependabot configuration and workflow (weekly grouped PRs)
    • Automate project‑board card creation by labeling or using the GitHub Projects API

  • Environment Management

    • Development (local with Ollama)
    • Staging (Cloudflare preview)
    • Production (mikepfunk.com)
    • Secrets management (Cloudflare Workers KV)

📊 Monitoring & Analytics

Phase 10: Observability

  • Sentry Error Tracking

    • Client-side error capture
    • Server-side error capture
    • Performance monitoring (Core Web Vitals)
    • User session replay

  • Search Analytics

    • Query success rates
    • Model performance comparison
    • User engagement metrics
    • Cost tracking per provider

🐛 Known Issues to Fix

Critical Bugs

  • Cloudflare build failing (.output/server directory)
  • Convex MCP Node.js API errors (resolved with dynamic imports)
  • TanStack devtools menu appearing (removed TanStackDevtools component)
  • CSRF 403 errors on /api/chat (created /api/csrf-token endpoint + useCsrfToken hook)
  • Infinite Ollama detection loop (fixed with useMemo + useRef guard)
  • Hydration warnings (removed suppressHydrationWarning, fixed ReactMarkdown plugin order)
  • ReactMarkdown build error (fixed remarkGfm in wrong plugin array)
  • Vite production build import errors (added .ts/.tsx extensions to all local imports)
  • ADD discriminator not implemented (built real adversarial validation with 5 parallel discriminators)
  • Model verification cache missing TTL (added 5-minute cache expiration)
  • MCP type bypasses with 'as any' (properly typed tool handlers)
  • ParallelModelOrchestrator Ollama-only (refactored to support all providers)
  • Auth not explicitly controlled (added VITE_DISABLE_AUTH env var)
  • tsconfig.json invalid ignoreDeprecations (removed deprecated option)
  • Missing useCsrfToken import extension (fixed for Vite SSR build)
  • ModelConfigManager connected to AI SDK (types exported, end-to-end verified)
  • No Convex schema for user data (comprehensive schema created)
  • Missing wrangler.toml configuration

Medium Priority

  • API keys in plain localStorage (need encryption) - DONE: Web Crypto API + Convex backup
  • No CSRF protection on some routes - DONE: HttpOnly cookies + X-CSRF-Token headers
  • Large bundle size (1.2MB main.js)
  • Missing TypeScript strict mode compliance

Low Priority

  • Improve error messages for failed searches
  • Add loading skeletons for chat messages
  • Optimize image assets
  • Add PWA support

📅 Timeline Estimate

Phase Duration Status
Phase 1: Infrastructure 1-2 days 🟢 95% Complete
Phase 2: Backend Services 2-3 days 🟢 100% Complete
Phase 3: Model Integration 2-3 days 🟢 95% Complete
Phase 4: Chat & Search 3-4 days 🟢 100% Complete
Phase 5: Memory System 3-4 days 🟢 80% Complete
Phase 6: OCR + Vision 2-3 days 🟢 80% Complete
Phase 7: Vector + Graph 2-3 days 🟢 90% Complete
Phase 7b: Provider Adapter 1-2 days 🟢 100% Complete
Phase 8: Testing 2-3 days 🟢 85% Complete
Phase 9: CI/CD 1-2 days 🟢 80% Complete
Phase 10: Monitoring 1-2 days 🟢 70% Complete
Phase 11: Query Enhancement 1-2 days 🟢 95% Complete
Phase 12: Caching 2-3 days 🟢 70% Complete

Total Estimated Time: 18-28 days

🎯 Success Criteria

  • Successfully deployed to mikepfunk.com on Cloudflare
  • Local Ollama models working without API keys (DONE: auto-detection at localhost:11434)
  • Cloud models (Anthropic, OpenAI) working with encrypted keys (DONE: Web Crypto API + Convex)
  • Chat interface with streaming responses (DONE: AgenticChat component)
  • Agentic search returning relevant results (Partial: backend pending)
  • Short-term memory (Convex) operational (DONE: schema created)
  • Long-term memory (S3/DynamoDB) operational (Pending: export functionality)
  • MCP server integration functional
  • All critical bugs fixed (DONE: devtools, CSRF, infinite loop)
  • CSRF protection enabled (DONE: HttpOnly cookies + headers)
  • Sentry tracking errors and performance (DONE: configured)
  • CodeRabbit reviewing PRs automatically

Human-in-the-Loop Learning Criteria

  • SegmentApprovalPanel allows approve/edit/reject workflow (DONE: Full interactive UI with confidence ratings)
  • SearchHistory displays past searches with filters (DONE: Pagination, quality filtering, statistics)
  • User approval rate >85% (measure AI segment quality) (Pending: Need production data)
  • User modification rate <20% (measure AI accuracy) (Pending: Need production data)
  • Search quality ADD score >0.80 (discriminator-based) (DONE: ADD discriminator functional)
  • Training data exported in JSONL format (DONE: OpenAI/Anthropic/Generic export formats)
  • SearchComparisonDashboard shows search results side-by-side (DONE: Full comparison with metrics)

📝 Next Immediate Actions

Completed ✔️

  • 1. Fix Cloudflare build (create .output/server directory in build script)
  • 2. Create Convex schemas for models, chat, search results (DONE: comprehensive schema with 15+ tables)
  • 3. Build chat interface component with streaming support (DONE: AgenticChat with CSRF)
  • 4. Test with Ollama local model first (no API key needed) (DONE: auto-detection working)
  • 5. Fix TanStack devtools menu appearing (DONE: removed component)
  • 6. Fix CSRF 403 errors (DONE: /api/csrf-token endpoint)
  • 7. Fix infinite Ollama detection loop (DONE: useMemo + useRef)
  • 8. Document complete system architecture (DONE: SYSTEM_ARCHITECTURE.md)
  • 9. Implement ADD discriminator (DONE: 5 parallel discriminators with adversarial detection)
  • 10. Build researcher-style results storage (DONE: ResearchStorage with annotations, indexing, 4 export formats)
  • 11. Fix type safety issues (DONE: removed 'as any', added cache TTL, proper MCP typing)
  • 12. Refactor ParallelModelOrchestrator (DONE: supports OpenAI, Anthropic, Google, Ollama, Azure)
  • 13. Add explicit auth control (DONE: VITE_DISABLE_AUTH with 3-tier behavior)
  • 14. Fix all build errors (DONE: Vite SSR imports, tsconfig, ReactMarkdown)
  • 15. Build SegmentApprovalModal.tsx (DONE: interactive segment control with QuerySegment types)
  • 16. Build SearchHistoryPage.tsx (DONE: browse/filter/export past searches)
  • 17. Export model types (DONE: ModelProvider, AvailableModels types exported)
  • 18. Production build passing (DONE: builds successfully, 751KB main.js, 376KB server.js)

In Progress 🔵

  • 11. Create /api/search/interactive - Segment proposal endpoint (Optional enhancement)
  • 12. Create /api/search/execute - Execute approved segments (Optional enhancement)

Recently Completed ✔️

  • 19. Build SearchHistory.tsx (DONE: Full history browser with pagination, filtering, statistics)
  • 20. Build SegmentApprovalPanel.tsx (DONE: Interactive approval UI with confidence ratings)
  • 21. Build ReasoningStepValidator.tsx (DONE: Step-by-step reasoning validation UI)
  • 22. Build DatasetExportDashboard.tsx (DONE: Export training data in OpenAI/Anthropic/Generic JSONL)
  • 23. Build SearchComparisonDashboard.tsx (DONE: Side-by-side search comparison with metrics)
  • 24. Create production routes (DONE: /history, /export, /comparison routes functional)
  • 25. Fix all build errors (DONE: Production build passes in 966ms)
  • 26. Create useCsrfToken hook (DONE: CSRF token management working)
  • 27. Document production status (DONE: PRODUCTION_STATUS.md created)
  • 28. Wire up saveSearch() in AgenticChat (DONE: Auto-saves after each search, lines 206-219)
  • 29. Add navigation links in Header (DONE: History, Comparison, Export links added, lines 88-125)
  • 30. Full integration complete (DONE: All features connected to backend, 100% operational)

Pending ⏳

  • 14. Create wrangler.toml for Cloudflare configuration
  • 15. Deploy to Cloudflare and test at mikepfunk.com
  • 16. Add training data export to S3 (JSONL format) - Convex export functional, S3 optional
  • 17. Initialize Convex with npx convex dev (if not already running)

Completed in This Session ✔️

  • Query Enhancement Pipeline (src/lib/query-enhancement/)

    • Spelling correction with common misspellings dictionary
    • Entity recognition for tech products, organizations, dates
    • Query expansion with synonyms
    • Context injection from user history
    • Language detection

  • Semantic Caching Layer (src/lib/semantic-cache/)

    • Vector-based query similarity matching (88% threshold)
    • Memory cache with LRU eviction
    • Cosine similarity for semantic matching
    • Cache hit/miss tracking with stats
    • Integrated into UnifiedSearchOrchestrator

  • CodeRabbit CI/CD Setup

    • .coderabbit.yaml with assertive profile
    • Path-specific review instructions
    • GitHub Actions workflow for CI/CD
    • Cloudflare Pages preview deployments

  • Observability Service (src/lib/observability/)

    • Distributed tracing with spans
    • Custom metrics (latency, tokens, quality)
    • Search trace recording
    • Model call trace recording
    • Integrated into search flow

  • Vector Storage (src/lib/vector-storage/)

    • In-memory fallback with cosine similarity
    • LanceDB support with dynamic import
    • CRUD operations with embeddings
    • Metadata filtering support
    • 5 tests passing

  • Knowledge Graph (src/lib/knowledge-graph/)

    • Entity extraction and normalization
    • Relationship mapping
    • Path finding between entities
    • Query expansion with graph context
    • Serialization for persistence
    • 5 tests passing

  • Translation Service (src/lib/translation/)

    • Language detection for 10 languages
    • Entity preservation during translation
    • Translation caching with TTL
    • 8 tests passing (2 minor failures on edge cases)

  • Model Routing (src/lib/model-routing/)

    • Query complexity classification
    • Cost-aware routing decisions
    • Fallback chain generation
    • Manual override support
    • Local model preference
    • 7 tests passing

💥 Recent Commits & Bug Fixes

Session 2024-01-XX: Critical Bug Fixes

Commit 1: Remove TanStack Devtools Menu

  • Issue: Unwanted settings panel ("General", "Default open", "Hide trigger") appearing on page
  • File: src/routes/__root.tsx
  • Changes:
    • Removed <TanStackDevtools /> component (lines 68-80)
    • Added suppressHydrationWarning to <body> tag (line 63)
    • Updated page title to "Agentic Search - The Future of Intelligent Search"
  • Result: Clean UI without devtools interference

Commit 2: Fix CSRF 403 Forbidden Errors

  • Issue: POST /api/chat failing with 403 due to missing CSRF token cookie
  • Root Cause: CSRF token cookie not being set on page load, but client trying to send immediately
  • Files Modified:
    1. Created: src/routes/api/csrf-token.ts (20 lines)
      • GET endpoint that generates CSRF token and sets HttpOnly cookie
    2. Modified: src/hooks/useCsrfToken.tsx (lines 35-82)
      • Added isInitialized state
      • Auto-fetches /api/csrf-token if cookie doesn't exist
      • Sets cookie server-side
    3. Modified: src/components/AgenticChat.tsx (lines 34, 43, 149, 338-340, 357)
      • Added isReady = !!csrfToken && !csrfError
      • Disabled textarea/submit until CSRF ready
      • Changed placeholder to "Initializing security..." when not ready
  • Flow:
    1. Page loads → hook checks for cookie
    2. No cookie → fetches /api/csrf-token
    3. Server sets HttpOnly cookie
    4. Hook reads cookie, sets csrfToken state
    5. isReady = true, chat enabled
    6. User sends message with X-CSRF-Token header
    7. Server validates cookie matches header
    8. Request succeeds
  • Result: CSRF protection working correctly, no more 403 errors

Commit 3: Fix Infinite Ollama Connection Detection Loop

  • Issue: http://localhost:11434/api/tags fetching repeatedly in infinite loop
  • Root Cause: modelOptions array recreated on every render, causing useEffect to re-run infinitely
  • File: src/components/EnhancedModelSelector.tsx
  • Changes:
    • Line 7: Added imports useMemo, useRef
    • Line 34: Added const hasDetected = useRef(false)
    • Line 37: Wrapped modelOptions in useMemo(() => [...], [])
    • Line 65: Added closing ], []) for useMemo
    • Lines 105-108: Added if (hasDetected.current) return; hasDetected.current = true; at start of useEffect
    • Line 118: Changed dependency array from [] to [modelOptions]
  • Result: Ollama detection runs exactly once per component mount, no infinite loops

Commit 4: Document Human-in-the-Loop Learning System

  • Created: docs/SYSTEM_ARCHITECTURE.md (644 lines)
  • Content:
    • Interactive segmentation workflow with user approval UI
    • Encrypted API key storage (Web Crypto API + Convex)
    • Search history browsing and result presentation
    • Comparison dashboard for side-by-side segment results
    • Training data collection and model fine-tuning pipeline
    • API endpoint specifications
    • UI mockups for SegmentApprovalModal and SearchHistoryPage
    • Success metrics and security considerations
  • Result: Complete system architecture documented for implementation

Commit 5: Update README.md and plan.md

  • Modified: README.md
    • Updated title to "Agentic Search Platform"
    • Added "Status: Production Ready" section
    • Documented all completed bug fixes
    • Listed human-in-the-loop features
    • Updated tech stack and key components
    • Added "Recent Bug Fixes" section with detailed solutions
  • Modified: docs/plan.md
    • Marked completed bug fixes as [x]
    • Updated "Next Immediate Actions" with completed items
    • Added "Human-in-the-Loop Learning Criteria" to success metrics
    • Split actions into Completed/In Progress/Pending sections
  • Result: Documentation fully reflects current system state

🔗 Resources