Proposal: Multi-Modal Ingestion & Retrieval (MVP)

[mukulpythondev]

Proposal: Multi‑Modal Ingestion & Retrieval (MVP)

Short summary
Add first‑class support in Rago for multi‑modal inputs (images, audio, and figures inside documents) so they can be indexed, embedded, and searched alongside text. This will enable users to query images/figures, search video/audio transcripts, and retrieve OCR'd text from screenshots — unlocking research workflows (slides, papers with diagrams, recorded lectures) that are common in open science.

---

Problem statement

Current Rago pipelines focus primarily on text extraction and text embeddings. Many research artifacts are multi‑modal: figures, tables embedded as images, screenshots of handwritten notes, recorded lectures, and short videos. Researchers often need to search across these modalities but are forced to manually transcribe, OCR, or ignore non‑text content.

Effect: reduced utility for research workflows, missed content (figures, charts, code screenshots), and friction for users who must pre‑process files outside Rago.

---

Proposed feature (MVP)

Implement a modular, opt‑in multi‑modal ingestion pipeline with three main capabilities:

1. Image OCR & embedding

   • Extract text from images (PDF figures, screenshots) using an OCR engine and emit text chunks to the existing text pipeline.
   • Generate image embeddings (optional) using a vision‑encoder model and store them alongside text embeddings in the vector DB (tagged as modal=image).

2. Audio transcription & embedding

   • Transcribe audio files (mp3, wav) using an ASR component (e.g., Whisper) and index the resulting transcript with timestamps.
   • Optionally embed short audio segments (or embeddings from transcript) and allow retrieval by time‑range or nearest transcript chunk.

3. Metadata & chunk tagging

   • During ingestion tag chunks with modal (text|image|audio), source_file, page (for PDFs), and time_range (for audio). This enables filtered semantic search (e.g., modal:image or type:figure).

MVP scope constraints

• Support OCR for common image formats and images embedded in PDFs.
• Support Whisper small/medium (configurable) for transcription (or provide adapter pattern so users plug their ASR).
• Store OCR text in existing chunk format so Rago's retriever and generator work unchanged.
• Provide configuration flags to enable/disable image/audio processing to avoid extra cost for users.

---

Why this matters / Impact

• Makes Rago usable for scientific content that contains figures, charts, and recorded talks.
• Aligns tightly with Open Science mission: improves discoverability of non‑text artifacts in research outputs.
• Differentiates Rago from text‑only RAG tools and expands potential user base (labs, professors, students).

---

Design & Implementation plan

Architecture

• New ingestion module rago.ingest.multimodal with three adapters: ocr_adapter, vision_embed_adapter, audio_transcribe_adapter.
• Reuse existing chunking + embedding pipeline: OCR/audio transcripts become text chunks and pass through the same embedding and indexing steps.
• Image embeddings stored as separate vectors but with cross‑references to OCR chunks (so either text or image result can be surfaced).

Implementation steps (MVP)

1. Add configuration options: enable_image_ocr, enable_image_embedding, enable_audio_transcription, audio_model.

2. Implement ocr_adapter:

   • Use an OCR library (adapter pattern). Provide two adapters: tesseract (local) and hf-ocr (adapter for Hugging Face OCR models).
   • Extract text and page coordinates (when available) and emit standard chunks.

3. Implement audio_transcribe_adapter:

   • Adapter to Whisper (local or API) that returns transcript + timestamps.
   • Chunk transcripts into suitable sizes for embedding.

4. Implement vision_embed_adapter (optional for MVP but strongly recommended):

   • Use a prebuilt vision encoder (e.g., CLIP via Hugging Face) to create image vectors.
   • Store with metadata and allow retrieval filtering by modal:image.

5. Update indexing to attach modal metadata and support filtering queries.

6. Add unit tests and end‑to‑end integration test (small PDF with figure + short audio file).

7. Add example notebook and docs showing how to enable and use the feature.

Backward compatibility

• Default: multi‑modal processing is off. No changes to current behavior unless enabled in config.
• Existing text pipelines remain unchanged.

---

API / CLI changes (suggested)

• Add flags to CLI/ingest API: --enable-image-ocr, --enable-image-embeddings, --enable-audio.
• In programmatic API, add IngestOptions struct with same booleans and model selection.

---

Libraries / Dependencies (suggested adapters)

• OCR: Tesseract (pytesseract wrapper) or Hugging Face Layout‑LM/TrOCR adapters
• ASR: OpenAI Whisper (local) or Whisper API adapter, or Hugging Face whisper models
• Vision embeddings: CLIP (Hugging Face sentence-transformers image models) or OpenAI image embeddings
• Vector DB: reuse existing interface (FAISS/Pinecone) — store an extra modal field in metadata

Note: Implementation should follow adapter pattern so maintainers who prefer not to pull heavy deps can still opt for minimal (OCR only) implementations.

---

Tests & Validation

• Unit tests for adapters (mocking OCR/ASR responses)
• Integration test: ingest a small PDF with a figure and an audio file, then run a sample query to ensure OCR text and transcript are retrievable.
• Performance test: measure additional ingestion time and memory for image/audio processing (document in PR).

---

Docs / Examples

• Add a multimodal.md docs page with usage examples and configuration.
• Provide a demo notebook examples/multimodal_demo.ipynb showing ingestion and queries for PDF+audio.

---

Timeline (if I implement)

• Week 1: Design + adapters skeleton + config options
• Week 2: OCR adapter (Tesseract) + unit tests
• Week 3: Audio transcription adapter + chunking + unit tests
• Week 4: Vision embeddings (CLIP) + integration test + docs + example notebook

(Adjustable based on maintainer feedback and reviewer availability.)

---

Request for maintainers

• Is this feature aligned with the project's goals and roadmap? (I noticed roadmap items about adding backends and declarative APIs.)
• Any preferred libraries/dependencies or style guidelines I should follow?
• Would maintainers prefer this split into separate PRs (ocr, audio, vision) or one larger PR?

If this looks good I can open a draft issue/PR and start implementing — I’d appreciate any pointers on preferred module names, coding conventions, and tests.

---

Thanks — I’m excited to contribute and can open a draft PR if maintainers agree on the approach.

[xmnlab]

I am working on a new API for rago. After that we could discuss about this.