rag_watch

RAG-based technology watch for cybersecurity and AI research.

Ingest papers and articles, build a semantic index, query with RAG, and generate trend reports.

Quickstart

Local Setup

# Install
pip install -e ".[dev]"

# Ingest arXiv papers
python -m rag_watch ingest --source arxiv --query "prompt injection" --days 30

# Ingest RSS feeds (configured in config.yaml)
python -m rag_watch ingest --source rss --query "security" --days 30

# Build FAISS index
python -m rag_watch build-index

# Ask a question (requires Ollama running)
python -m rag_watch ask --question "What are the latest prompt injection defenses?" --k 8

# Compare two time periods
python -m rag_watch compare \
  --topic "LLM security" \
  --from 2025-07-01 --to 2025-12-31 \
  --baseline-from 2025-01-01 --baseline-to 2025-06-30

# Generate monthly report
python -m rag_watch report --month 2026-02

# Generate research questions
python -m rag_watch questions --topic "prompt injection"

Docker Setup

# Start Ollama + rag_watch
docker compose -f docker/docker-compose.yaml up -d ollama

# Pull a model
docker compose -f docker/docker-compose.yaml exec ollama ollama pull llama3.1:8b

# Run rag_watch commands
docker compose -f docker/docker-compose.yaml run rag-watch ingest --source arxiv --query "prompt injection" --days 7
docker compose -f docker/docker-compose.yaml run rag-watch build-index
docker compose -f docker/docker-compose.yaml run rag-watch ask --question "What is prompt injection?"

Configuration

Edit config.yaml to customize:

Change embedding model

embedding:
  model_name: intfloat/e5-large-v2  # or all-MiniLM-L6-v2
  dimension: 1024                    # match the model's output dimension

Change LLM provider

The LLMClient interface is abstract. Currently Ollama is implemented.

llm:
  provider: ollama
  model_name: llama3.1:8b
  base_url: http://localhost:11434

To add another provider (e.g., OpenAI), implement LLMClient in rag_watch/rag/llm_client.py and register it in create_llm_client().

Add RSS feeds

ingest:
  rss_feeds:
    - name: schneier
      url: https://www.schneier.com/feed/atom
    - name: my-custom-feed
      url: https://example.com/feed.xml

Project Structure

rag_watch/
  config.py              # Configuration dataclasses + YAML loader
  models.py              # Document, Chunk, RetrievedChunk models
  cli.py                 # CLI entry point
  utils/                 # Hashing, JSONL, logging helpers
  ingestion/             # BaseIngestor + arXiv/RSS connectors
  preprocessing/         # Text cleaning + chunking
  index/                 # Embedding service + FAISS store
  rag/                   # LLM client, prompts, retriever, questions
  analytics/             # Emerging terms, clustering, report generation
tests/                   # Unit tests
data/                    # Ingested documents, chunks, FAISS index
reports/                 # Generated reports
docker/                  # Dockerfile + docker-compose

Reproducibility

Each run is identified by a run_id (hash of config). All artifacts are saved:

• data/documents.jsonl - raw ingested documents
• data/chunks.jsonl - chunked documents
• data/index/faiss.index - FAISS vector index
• reports/*.md - generated reports

Running Tests

pytest