🩺 Health Records RAG Copilot

A fully local, zero-cost RAG (Retrieval-Augmented Generation) system for querying health records and clinical guideline documents — built with only open-source, self-hostable tools. No paid APIs, no cloud accounts, no signups required to run it end-to-end.

Why this exists: most RAG portfolio projects wrap OpenAI/Claude behind a vector search and call it done. This project instead treats the two hardest real engineering problems in RAG head-on: (1) retrieval quality (hybrid search + reranking, measured with an eval harness, not vibes) and (2) structured-output reliability from a local, non-function-calling-native model (retry/validation logic, measured as a first-class metric).

---

Tech Stack (all free / local / open-source)

Component	Tool	Why
LLM	Ollama running llama3.1:8b or qwen2.5:7b	Local inference, zero API cost, zero API key
Embeddings	sentence-transformers (BAAI/bge-small-en-v1.5)	Runs on CPU, no external calls
Reranker	cross-encoder/ms-marco-MiniLM-L-6-v2	Local, CPU-friendly, big retrieval quality boost
Vector DB	Qdrant (Docker)	Open-source, self-hosted, no cloud account
Keyword search	rank-bm25	Classic sparse retrieval, complements dense vectors
Backend	FastAPI	Async, typed, easy to reason about
Metadata/logs	SQLite	Zero-config, file-based
Frontend	Streamlit	Fastest path to a usable chat UI + dashboard

---

Architecture

flowchart TD
    A[User uploads PDF/CSV/TXT] --> B["/ingest endpoint"]
    B --> C[Parse + Chunk<br/>500 tokens, 50 overlap]
    C --> D[Embed locally<br/>bge-small-en-v1.5]
    D --> E[(Qdrant<br/>vector index)]
    C --> F[(SQLite<br/>chunk text + metadata)]

    G[User asks a question] --> H["/query endpoint"]
    H --> I[Vector search<br/>Qdrant]
    H --> J[BM25 keyword search<br/>rank_bm25]
    I --> K[Weighted score fusion]
    J --> K
    K --> L[Cross-encoder reranking]
    L --> M[Top-k chunks]
    M --> N[Local LLM via Ollama<br/>forced JSON schema]
    N --> O{Valid JSON?}
    O -->|No, retry <= 3x| N
    O -->|Yes| P[Structured answer<br/>+ findings]
    P --> Q[Streamlit UI:<br/>answer + sources + feedback]
    P --> F2[(SQLite<br/>query logs)]

Loading

---

Quickstart

1. Prerequisites

• Docker + Docker Compose
• Ollama installed on your host machine (not in Docker — see note below)
• Python 3.11+ (only needed if you run the frontend/eval scripts outside Docker)

2. Pull a local model

ollama pull llama3.1:8b
# Lower-spec machine (< 8GB free RAM)? Use a smaller model instead:
# ollama pull qwen2.5:3b
# ollama pull phi3:mini

3. Start Qdrant + backend

docker-compose up --build

This starts:

• Qdrant on localhost:6333
• FastAPI backend on localhost:8000 (docs at localhost:8000/docs)

The backend reaches your host's Ollama instance via host.docker.internal — already configured in docker-compose.yml, no extra setup needed.

4. Ingest the sample data

python scripts/ingest_sample_data.py

This loads 5 synthetic lab reports, 3 mock clinical guidelines, and a wearable CSV export from sample_data/ — enough for an instant, realistic demo.

5. Run the frontend

cd frontend
pip install -r requirements.txt
streamlit run streamlit_app.py

Open the URL Streamlit prints (usually localhost:8501) and start asking questions like:

• "What was patient PT-1001's ferritin level and what does it indicate?"
• "What first-line treatment is recommended for iron-deficiency anemia?"

6. Run the evaluation harness

cd eval
pip install -r requirements.txt
python run_eval.py

This overwrites eval/eval_report.md with real metrics against the 25-question eval set — retrieval precision/recall, answer faithfulness, and (importantly) local-model JSON reliability.

---

Minimum Hardware Notes

• llama3.1:8b / qwen2.5:7b: recommend 8GB+ free RAM, runs on CPU but is noticeably faster with any GPU (even a modest one) or Apple Silicon.
• Lower-spec machines: use qwen2.5:3b or phi3:mini — both are ~2-4x faster at the cost of somewhat weaker JSON-schema adherence (see eval notes below).
• Embedding + reranking models are small (<150MB combined) and run comfortably on CPU regardless of which LLM you choose.

---

A Note on Ollama and Docker

Ollama is intentionally not included as a docker-compose service. Running it inside Docker typically loses access to GPU/Metal acceleration on most laptops and adds friction to model pulls. Instead: install Ollama natively on your host, and the backend container reaches it at http://host.docker.internal:11434 (already wired into docker-compose.yml).

---

Evaluation Results

See eval/eval_report.md for the full generated report (25 hand-written Q&A pairs against the sample documents). The report is regenerated by run_eval.py and includes:

• Retrieval Precision@5 / Recall@5 — did hybrid search + reranking surface chunks from the correct source document?
• Answer faithfulness — embedding cosine similarity between each generated finding and the chunk it cites, as a rough hallucination check.
• JSON-valid-on-first-try rate — the percentage of queries where the local model produced schema-valid JSON without needing a retry. This is the metric that most separates "demo that works once" from "system I'd trust in production," and it's rarely reported in RAG tutorials that assume a hosted, function-calling-native API.
• Average retries needed and average end-to-end latency.

Run it yourself — numbers will vary by model choice, hardware, and prompt tuning, which is exactly the point.

---

What I'd Improve With a Hosted Model

Being upfront about tradeoffs (this is more useful to a reviewer than pretending there are none):

1. Native function calling. Hosted APIs like Claude/GPT support structured tool-calling natively, which is far more reliable than prompting a local model into JSON and validating after the fact. The retry logic here is a direct mitigation for this gap — the JSON-validity metric quantifies exactly how much it matters.
2. Reasoning depth on ambiguous questions. Smaller local models (7-8B) are noticeably weaker than frontier hosted models at multi-hop reasoning across several retrieved chunks. A hosted model would likely improve faithfulness scores on harder eval questions.
3. Latency. Local CPU inference is slower than hosted API calls, especially for longer contexts. A hosted model (or a GPU-accelerated local deployment) would cut end-to-end latency significantly.
4. Fallback strategy in production: a realistic production version of this system would likely route simple queries to the local model (free, private) and escalate ambiguous/high-stakes queries to a hosted model with a human confirming the tradeoff — a hybrid-cost architecture, not an all-or-nothing choice.

---

API Reference

Endpoint	Method	Description
/ingest	POST	Upload a PDF/CSV/TXT/MD file, chunk + embed + index it
/query	POST	{"question": str, "top_k": int} — hybrid retrieval + structured LLM answer
/feedback	POST	{"query_id": str, "rating": "up"|"down", "comment": str}
/stats	GET	Aggregated latency, JSON-reliability, and usage dashboard
/health	GET	Liveness check

Interactive API docs available at http://localhost:8000/docs once the backend is running.

---

Project Structure

health-records-rag-copilot/
├── backend/
│   ├── app/
│   │   ├── main.py          # FastAPI endpoints
│   │   ├── ingestion.py     # parse -> chunk -> embed -> upsert
│   │   ├── retrieval.py     # hybrid search + reranking
│   │   ├── generation.py    # structured LLM output + retry/validation
│   │   ├── models.py        # Pydantic schemas
│   │   ├── db.py            # SQLite layer
│   │   └── config.py        # all tunables, env-overridable
│   ├── requirements.txt
│   └── Dockerfile
├── frontend/
│   └── streamlit_app.py     # chat UI, ingest UI, dashboard
├── eval/
│   ├── eval_set.json        # 25 hand-written Q&A pairs
│   ├── run_eval.py          # eval harness (precision/recall/faithfulness/JSON reliability)
│   └── eval_report.md       # generated report
├── sample_data/             # synthetic lab reports, guidelines, wearable CSV
├── scripts/
│   └── ingest_sample_data.py
├── docker-compose.yml
├── .env.example
└── README.md

---

Disclaimer

All patient data in sample_data/ is entirely synthetic/fictional, generated for demo purposes. Clinical guideline documents are mock documents written for this project and are not official medical guidance. This project is a software engineering portfolio piece, not a medical product.