Skip to content

README.md

Latest commit

 

History

History
196 lines (142 loc) · 6.11 KB

README.md

File metadata and controls

196 lines (142 loc) · 6.11 KB
Raw

RAGLens

RAGLens is a full-stack Retrieval-Augmented Generation (RAG) evaluation and observability platform.

It is designed around three service responsibilities:

rag-api answers questions.
eval-api decides whether those answers are good.
dashboard explains what happened.

Repository structure

apps/
  rag-api/       Fastify + TypeScript service for ingestion, retrieval, answers, and traces
  eval-api/      FastAPI + Python service for datasets, scoring, comparisons, and CI gates
  dashboard/     Next.js dashboard for document, trace, and eval inspection

infra/
  migrations/    PostgreSQL and pgvector schema bootstrap
  seed/          Controlled seed documents and eval datasets

packages/
  shared/        Shared contracts that are safe to reuse across TypeScript workspaces

docs/            Source-of-truth project documentation in Markdown
docs-site/       Astro static documentation site that renders docs from docs/

Current bootstrap scope

The current implementation provides:

  • rag-api health endpoint at GET /api/v1/health
  • eval-api health endpoint at GET /api/v1/health
  • rag-api Markdown document ingestion, retrieval, query, and trace endpoints
  • rag-api advanced retrieval modes (vector, keyword, hybrid, hybrid_reranked) with query rewriting, context packing, and configurable reranker adapter controls
  • rag-api provider runtime adapters for deterministic/OpenAI/OpenRouter/Anthropic/Ollama query answering
  • rag-api provider token usage + optional pricing configuration for estimated cost telemetry
  • rag-api request correlation and structured provider-failure envelopes with x-request-id
  • Postgres-backed document storage with rag.documents and rag.document_chunks
  • Postgres-backed eval persistence for datasets, eval runs, case results, comparisons, and CI gate runs
  • eval-api request correlation and structured error envelopes with x-request-id
  • eval runner retries, timeout mapping, cost/max-case guardrails, and partial failure persistence
  • seed corpus and golden eval fixture under infra/seed
  • dashboard document list/detail screens for corpus inspection
  • dashboard devtools page at /devtools for manual rag-api and eval-api request inspection
  • PostgreSQL + pgvector via Docker Compose
  • CI checks for Node, Python, docs, Docker Compose config, and Docker Compose image builds

Runtime planning

RAGLens is planned to support both cloud and local model runtimes through provider-neutral interfaces and named model profiles.

The current implementation is not yet at local LLM execution. Local runtime support is planned to enter during the embeddings, retrieval, answer generation, trace, and eval phases rather than as a late rewrite.

Key docs:

Initial planned local profile:

local-balanced
  chat provider: ollama
  chat model: qwen3:8b
  embedding provider: ollama
  embedding model: nomic-embed-text
  reranker provider: none
  retrieval mode: vector

Local setup

Copy the example environment file:

cp .env.example .env

Install Node dependencies:

npm install

Install the Python eval API locally when working outside Docker:

cd apps/eval-api
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Start the local infrastructure and services:

docker compose up --build

Service URLs:

Dashboard: http://localhost:3000
rag-api:   http://localhost:8000/api/v1/health
eval-api:  http://localhost:8001/api/v1/health
Postgres:  localhost:5432

Seed ingestion

Load the controlled seed corpus into rag-api:

npm --workspace apps/rag-api run seed:documents

Then inspect indexed documents in the dashboard:

http://localhost:3000/documents

You can also upload ad-hoc Markdown/text files from the dashboard UI:

http://localhost:3000/documents/upload

See docs/seed-ingestion.md for the full workflow and troubleshooting notes.

Local eval workflow

  1. docker compose up --build
  2. npm --workspace apps/rag-api run seed:documents
  3. Open http://localhost:3000/datasets
  4. Create a dataset
  5. Add test cases
  6. Open http://localhost:3000/eval-runs/new
  7. Create an eval run
  8. Open eval run detail
  9. Execute run
  10. Inspect case results

/devtools remains available for low-level API inspection, but normal eval execution should now run through dashboard datasets and eval run pages.

Dashboard shell workflow

The dashboard now uses a persistent application shell:

  • Left sidebar grouped by product area: Overview, Corpus, Evaluation, Operations
  • Sticky topbar with route-aware breadcrumb, title, and page actions
  • Content area for page-specific cards, forms, and detail panels

Recommended navigation flow:

  1. Open Overview (/) for service and workflow context
  2. Inspect corpus in Documents (/documents) and Retrieval (/retrieval)
  3. Build datasets in Datasets (/datasets)
  4. Create and execute eval runs from Eval Runs (/eval-runs, /eval-runs/new)
  5. Inspect run case results and comparisons (/comparisons/*)
  6. Use Devtools (/devtools) only for direct API debugging

Development commands

npm run typecheck
npm run lint
npm run test
npm run build

Docs commands:

npm run docs:dev
npm run docs:check
npm run docs:build
npm run docs:preview

Documentation authoring

  1. Add or update Markdown files in docs/.
  2. Ensure each file has frontmatter fields: title, description, order, section, status.
  3. Re-run npm run docs:build to validate and generate static output.

Astro build output is generated at docs-site/dist/.