Bring-your-own-corpus: HTTP API for document ingestion

[adaamko]

Summary

Extend the FastAPI server with endpoints for uploading and indexing documents at runtime, so users can build a verbatim RAG service over their own corpus without writing Python code.

Motivation

The current API exposes query and transform endpoints, but corpus management requires direct use of the Python library. An ingestion API would unlock use cases where documents arrive dynamically (e.g., nightly syncs, user uploads).

Proposed endpoints

POST /api/ingest          — upload and index a document (PDF, markdown, plain text)
DELETE /api/documents/{id} — remove a document from the index
GET  /api/documents        — list indexed documents

POST /api/ingest request:

• file (multipart) or url (string) — the document source
• title (string) — document title
• metadata (JSON object, optional)

Response: { "doc_id": "...", "chunks_indexed": 42 }

Notes

• Document processing should reuse the existing DocumentProcessor (docling + chonkie)
• The index is already backed by LocalMilvusStore; no new storage layer needed
• Authentication / API key gating is out of scope for this issue