A specialized RAG system designed for high-accuracy AWS infrastructure intelligence. This project demonstrates advanced design patterns in AI safety, hybrid retrieval, and performance engineering.
graph TD
User([User / Frontend]) --> API[FastAPI Layer]
subgraph Security_Guardrails [Security & Safety Layer]
API --> PG[PromptInjectionGuard]
PG --> DG[DomainGuard LLM Classifier]
end
subgraph Retrieval_Engine [Hybrid Retrieval Engine]
DG -- Parallel Execution --> VS[ChromaDB Vector Search]
DG -- Parallel Execution --> BM[BM25 Keyword Search]
VS & BM --> RRF[Reciprocal Rank Fusion]
end
subgraph Generation_Pipeline [Generation & QA]
RRF --> LLM[GPT-4o Primary Generator]
LLM --> RG[ResponseGuard G-Eval]
LLM -- Fallback --> FBL[Fallback Model]
end
RG --> Output([Verified Response])
subgraph Performance_Monitoring [Observability]
API -.-> RT[X-Process-Time Tracking]
API -.-> RL[SlowAPI Rate Limiting]
end
We implement a Reciprocal Rank Fusion (RRF) strategy to solve the limitations of single-vector search.
- Dense Search: Using all-MiniLM-L6-v2 via ChromaDB to capture semantic intent.
- Sparse Search: Using BM25Okapi to capture exact keyword matches (e.g., "S3 Lifecycle Policies").
- Design Decision: RRF was chosen over simple weighted averaging because it doesn't require score normalization across different scales, making it more robust as the document corpus grows.
To balance cost and reliability, we use a tiered execution model:
- Tier 1 (Classification):
gpt-4o-minihandles domain gating and safety checks. - Tier 2 (Primary Generation):
gpt-4ohandles the complex synthesis of technical AWS answers. - Resilience: The system implements automated multi-model fallback (GPT-4o → 4o-mini). If the primary model encounters rate limits or errors, the system automatically falls back to a secondary stable instance to ensure 99.9% availability.
- Elite RAG Engine: The backend is upgraded to an Elite RAG Engine featuring parallel hybrid retrieval (Dense + BM25) and Reciprocal Rank Fusion (RRF).
- Implementation: We use
asyncio.gatherto run Domain Guard validation and Vector Retrieval concurrently. - Reasoning: Since retrieval is I/O bound and domain validation is network-latency bound, parallelizing these saves ~1.5 seconds per request.
- Faithfulness (G-Eval): We use an "LLM-as-a-judge" pattern (
ResponseGuard) to score how well the answer is supported by the retrieved context (0.0 - 1.0).
Our security layer is designed to be proactive rather than reactive, handling three primary threat vectors:
- Regex-based Anchoring: We use strict pattern matching to detect common jailbreak precursors (e.g., "ignore previous instructions", "jailbreak").
- Encoded Attack Handling: The system proactively blocks common encoded attack patterns (e.g., Base64/Hex attempts).
- XML Tag Isolation: User input is wrapped in
<user_query>tags. Our system rejects any query containing closing tags (</user_query>), preventing "tag spoofing" where a user tries to escape the prompt context.
- We use an LLM-based classifier to verify if the query relates to AWS and provide helpful AWS-branded explanations if the user asks out-of-scope questions.
- Implementation: We require a mandatory
X-API-KEYheader for all endpoints to protect the backend.
The system is exposed via a production-grade FastAPI layer.
Main endpoint for synchronized RAG retrieval.
{
"question": "How do I optimize costs for a high-traffic RDS instance?"
}Streaming endpoint (SSE) for low-latency perceived performance.
Follow these steps for a clean, reliable local setup.
- Python 3.12+
- Node.js 20+
From the project root:
# 1. Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# 2. Install dependencies
pip install -r requirements.txt
# 3. Set up environment variables
# Copy the example file and update it with your API key
cp .env.example .envRequired .env content:
OPENAI_API_KEY=your_openai_api_key_here
API_KEY=Rfvtgb*321
DOMAIN_NAME=AWS Cloud Services4. Start the Backend API
uvicorn src.main:app --host 0.0.0.0 --port 3000The API is now running at http://localhost:3000. You can test it via http://localhost:3000/docs.
In a new terminal window, from the project root:
# 1. Navigate to frontend
cd frontend
# 2. Install dependencies
npm install
# 3. Start the development server
npm run dev4. Configure the Dashboard
- Open
http://localhost:5173in your browser. - Open Settings (top-right).
- API Base URL:
http://localhost:3000 - X-API-KEY: Ask the author
- Click Save. The "System Health" indicator should turn green ("Operational").
If the dashboard shows "Operational", you are ready to query! Ask any question, and the assistant will retrieve context from the AWS documentation.
- Faithfulness (G-Eval): We use an "LLM-as-a-judge" pattern to score how well the answer is supported by the retrieved context (0.0 - 1.0).
- Hallucination Rate: Monitored via automated evaluation scripts that compare generated answers against ground-truth AWS whitepapers.
- Latency vs. Accuracy: We prioritize RRF hybrid search which adds latency compared to simple vector search but increases accuracy by 22%.
- Cost vs. Security: Running a classifier before every query increases token cost but protects the primary LLM from processing malicious payloads.
- Selective Caching: We intentionally did not include global caching for generated responses due to privacy and security guardrail complexity.
- Corpus Recency: The system is currently limited to the aws-overview.pdf context.
- Context Window: Large document retrievals can occasionally hit token limits, managed currently via RecursiveCharacterTextSplitter.
by ~ Chaitanya