KX-nvidia-rag-blueprint

NVIDIA RAG Blueprint — KX fork

This fork supports KDB.AI and KDB-X as vector databases. The upstream NVIDIA RAG Blueprint supports Milvus, Elasticsearch, and KDB.AI as vector stores. This fork (kxdev/.../kx-nvidia-rag) specializes the Helm chart for KX vector databases — Milvus and Elasticsearch chart deps and overlays have been removed. Python implementations under src/nvidia_rag/utils/vdb/{milvus,elasticsearch}/ remain in the codebase (unused at deploy time). The Helm chart deploys either a KDB.AI-backed stack or a KDB-X-backed stack (connecting to a customer-managed bare kdb+ endpoint via IPC).

Also adds: NeMo Guardrails Helm support (orchestrator + content-safety + topic-control NIMs) — not available in upstream's Helm chart, only their docker-compose. See docs/nemo-guardrails.md.

Retrieval-Augmented Generation (RAG) combines the reasoning power of large language models (LLMs) with real-time retrieval from trusted data sources. It grounds AI responses in enterprise knowledge, reducing hallucinations and ensuring accuracy, compliance, and freshness.

Overview

The NVIDIA RAG Blueprint is a reference solution and foundational starting point for building Retrieval-Augmented Generation (RAG) pipelines with NVIDIA NIM microservices. It enables enterprises to deliver natural language question answering grounded in their own data, while meeting governance, latency, and scalability requirements. Designed to be decomposable and configurable, the blueprint integrates GPU-accelerated components with NeMo Retriever models, Multimodal and Vision Language Models, and guardrailing services, to provide an enterprise-ready framework. With a pre-built reference UI, open-source code, and multiple deployment options — including local docker (with and without NVIDIA Hosted endpoints) and Kubernetes — it serves as a flexible starting point that developers can adapt and extend to their specific needs.

Key Features

Data Ingestion

• Multimodal content extraction - Documents with with text, tables, charts, infographics, and audio. For the full list of supported file types, see [NeMo Retriever Extraction Overview](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/).
• Custom metadata support

Search and Retrieval

• Multi-collection searchability
• Hybrid search with dense and sparse search
• Reranking to further improve accuracy
• GPU-accelerated Index creation and search
• KDB.AI vector database with cuVS / CAGRA GPU acceleration; KDB-X (bare kdb+ BYO endpoint) also supports GPU CAGRA via kx.cuvs (set KDBX_USE_CUVS=1)

Query Processing

• Query decomposition
• Dynamic filter expression creation

Generation and Enrichment

• Opt-in for Multimodal and Vision Language Model Support in the answer generation pipeline.
• Document summarization
• Improve accuracy with optional reflection
• Optional programmable guardrails for content safety

Evaluation

• Evaluation scripts (RAGAS framework)

User Experience

• Sample user interface
• Multi-turn conversations
• Multi-session support

Deployment and Operations

• Docker Compose for local development
• Kubernetes/Helm for production deployments
• Amazon EKS with cloud-hosted NVIDIA AI endpoints
• Telemetry and observability
• Decomposable and customizable
• NIM Operator support
• Python library mode support
• OpenAI-compatible APIs

Software Components

The RAG blueprint is built from the following complementary categories of software:

• NVIDIA NIM microservices – Deliver the core AI functionality. Large-scale inference (e.g.for example, Nemotron LLM models for response generation), retrieval and reranking models, and specialized extractors for text, tables, charts, and graphics. Optional NIMs extend these capabilities with OCR, content safety, topic control, and multimodal embeddings.

• The integration and orchestration layer – Acts as the glue that binds the system into a complete solution.

This modular design ensures efficient query processing, accurate retrieval of information, and easy customization.

NVIDIA NIM Microservices

• Response Generation (Inference)

  • NVIDIA NIM llama-3.3-nemotron-super-49b-v1.5

• Retriever and Extraction Models

  • NVIDIA NIM llama-nemotron-embed-1b-v2
  • NVIDIA NIM llama-nemotron-rerank-1b-v2
  • NeMo Retriever Page Elements NIM
  • NeMo Retriever Table Structure NIM
  • NeMo Retriever Graphic Elements NIM
  • PaddleOCR NIM

• Optional NIMs

  • Llama 3.1 NemoGuard 8B Content Safety NIM
  • Llama 3.1 NemoGuard 8B Topic Control NIM
  • Nemotron-Nano-12b-v2-VL NIM
  • NeMo Retriever Parse NIM
  • NeMo Retriever OCR NIM (Early Access)
  • llama-nemotron-embed-vl-1b-v2

Integration and orchestration layer

• RAG Orchestrator Server – Coordinates interactions between the user, retrievers, vector database, and inference models, ensuring multi-turn and context-aware query handling. This is LangChain-based.

• Vector Database – Stores and searches embeddings at scale with high-performance indexing and retrieval.

  • KDB-X (default & recommended) – Default vector database for this fork (CAGRA / cuVS GPU by default). Connects to a customer-managed bare kdb+ endpoint via IPC — point the adapter at it with APP_VECTORSTORE_URL (e.g. http://my-kdbx-host:5000); the host:port is parsed from that URL. (KDBX_LISTEN_PORT is the server-side kdb+ listen port, not a client connection setting.) The KDB-X process loads its .rag.* server definitions from kdbx-init.q at q startup (the in-cluster test pod does this automatically via its entrypoint; a customer-managed KDB-X loads it itself — the adapter no longer pushes them on connect). Supports CPU HNSW and GPU CAGRA (via kx.cuvs). See docs/change-vectordb-kdbx.md for configuration and all KDBX_* env vars, and the step-by-step EKS + KDB-X cuVS (GPU CAGRA) setup runbook for a full from-scratch EKS deployment.
  • KDB.AI CuVS (supported alternative) – High-performance vector store built on kdb+, with GPU acceleration via NVIDIA cuVS. Select it by layering the KDB.AI overlay (-f deploy/EKS/rag-values-kdbai.yaml) — it ships public images and works out of the box.
  • Upstream NVIDIA RAG Blueprint also supports Milvus and Elasticsearch; that support has been removed from this Helm chart (Python implementations remain in the codebase for reference).

• NeMo Retriever Extraction – A high-performance ingestion microservice for parsing multimodal content. For more information about the ingestion pipeline, see NeMo Retriever Extraction Overview

• RAG User Interface (rag-frontend) – A lightweight user interface that demonstrates end-to-end query, retrieval, and response workflows for developers and end users. For more information, refer to RAG UI.

Technical Diagram

The following image represents the architecture and workflow.

Workflow

The following is a step-by-step explanation of the workflow from the end-user perspective:

1. Data Ingestion & Extraction Pipeline – Multimodal enterprise documents (text, images, tables, charts, infographics, and audio) are ingested.

2. User Query – The user interacts with the system through the UI or APIs, submitting a question. An optional NeMo Guardrails module can filter or reshape the query for safety and compliance before it enters the retrieval pipeline.

3. Query Processing – The query is processed by the Query Processing service, which may also leverage reflection (an optional LLM step) to improve query understanding or reformulation for better retrieval results.

4. Retrieval from Enterprise Data – The processed query is converted into embeddings using NeMo Retriever Embedding and matched against enterprise data stored in a cuVS accelerated Vector Database (CuVS) and associated object store(minIO). Relevant results are identified based on similarity.

5. Reranking for Precision – An optional NeMo Retriever Reranker reorders the retrieved passages, ensuring the most relevant chunks are selected to ground the response.

6. Response Generation – The selected context is passed into the LLM inference service (e.g., Llama Nemotron models). An optional reflection step can further validate or refine the answer against the retrieved context. Guardrails may also be applied to enforce safety before delivery.

7. User Response – The generated, grounded response is sent back to the user interface, often with citations to retrieved documents for transparency.

Minimum System Requirements

Hardware Requirements

The blueprint offers two primary modes of deployment. By default, it deploys the referenced NIM microservices locally. Each method lists its minimum required hardware. This will change if the deployment turns on optional configuration settings.

Docker

• 2x RTX PRO 6000
• 2x H100
• 3x B200
• 3x A100 SXM

Kubernetes

• 8x RTX PRO 6000
• 8x H100-80GB
• 9x B200
• 9x A100-80GB SXM
• 4x H100 (with Multi-Instance GPU / DRA with NIM Operator)

Tip

The blueprint allows for use of NVIDIA cloud-hosted endpoints, in which case the GPU requirements are significantly reduced — only GPUs for the ingestion NIMs and vector database are needed.

For detailed per-component requirements, see Minimum System Requirements.

OS Requirements

• Ubuntu 22.04 OS

Deployment Options

• Docker
• Kubernetes

Get Started With NVIDIA RAG Blueprint

KDB-X is the default and recommended vector database for this fork (GPU CAGRA via NVIDIA cuVS). For a cloud deployment, follow the step-by-step EKS + KDB-X cuVS runbook; for local evaluation or a self-hosted Kubernetes cluster, see the KDB-X deployment guide (covers Docker Compose and Helm). KDB.AI with NVIDIA cuVS remains supported as an alternative (public images, 90-day trial license) — see Alternative Vector Databases.

Deployment Options

The first three rows (KDB-X) are the recommended paths; the KDB.AI rows remain supported as an alternative.

Deployment	Vector Database	LLM	Documentation
Amazon EKS — step-by-step (Recommended)	KDB-X cuVS / GPU CAGRA (A10G)	NVIDIA Cloud Endpoints	Runbook
Kubernetes/Helm (Recommended)	KDB-X (BYO kdb+ endpoint)	Self-hosted NIM	Guide
Docker Compose — local eval	KDB-X	Self-hosted NIM or NVIDIA Cloud	Guide
Docker Compose	KDB.AI with cuVS (alternative)	Self-hosted NIM	Guide
Docker Compose	KDB.AI with cuVS (alternative)	NVIDIA Cloud Endpoints	Guide
Kubernetes/Helm	KDB.AI with cuVS (alternative)	Self-hosted NIM	Guide
Kubernetes with NIM Operator	KDB.AI with cuVS (alternative)	Self-hosted NIM	Guide

Sizing — dev vs. production. The EKS runbook uses a dev / evaluation GPU sizing (9× g5.2xlarge / A10G + the NGC-hosted LLM) so you can exercise the full pipeline cheaply. For preprod/production, self-host the LLM and size the GPU fleet per Minimum System Requirements (Kubernetes: 8× H100-80GB, 8× RTX PRO 6000 Blackwell, 9× A100-80GB SXM, or 9× B200).

LLM Hosting Options

The RAG Blueprint supports two options for LLM inference. Choose based on your infrastructure and requirements:

Option 1: Cloud-Hosted (NVIDIA API Endpoints) – No local GPU required for LLM

Use NVIDIA's cloud-hosted API endpoints for LLM inference. This is ideal when you want to minimize local GPU requirements or get started quickly.

Requirements:

• NVIDIA NGC API key (get one free)
• Internet connectivity to integrate.api.nvidia.com

Configuration:

# Set your NGC API key
export NGC_API_KEY="nvapi-your-key-here"
export NVIDIA_API_KEY="${NGC_API_KEY}"

# Configure LLM endpoint (in your .env or values file)
APP_LLM_SERVERURL="https://integrate.api.nvidia.com/v1"
APP_LLM_MODELNAME="nvidia/llama-3.3-nemotron-super-49b-v1.5"

Pros: No GPU required for LLM, faster setup, always up-to-date models Cons: Requires internet, API usage costs, data leaves your infrastructure

Option 2: Self-Hosted (Local NIM Containers) – Full control, air-gapped capable

Deploy the LLM as a local NIM container for complete control over your infrastructure. This is ideal for air-gapped environments, data privacy requirements, or high-throughput production workloads.

Requirements:

• GPU (for full self-hosted stack): One of the following configurations:
  • 2x H100
  • 3x B200
  • 3x A100 SXM
  • 2x RTX PRO 6000
  • See Minimum System Requirements for full details
• LLM-specific requirements: Refer to the NVIDIA NIM LLM Support Matrix
• Storage: ~50GB for model weights (cached in ~/.cache/nim)
• NVIDIA NGC API key (for downloading NIM containers)
• nvidia-container-toolkit installed

Configuration:

# Set your NGC API key
export NGC_API_KEY="nvapi-your-key-here"

# Configure LLM endpoint (points to local container)
APP_LLM_SERVERURL="http://nemollm-inference:8000/v1"
APP_LLM_MODELNAME="nvidia/llama-3.3-nemotron-super-49b-v1.5"

# GPU assignment for LLM (example: GPUs 0 and 1)
LLM_MS_GPU_ID=0,1

Pros: Data stays local, no per-query costs, works air-gapped, predictable latency Cons: Significant GPU investment, model management overhead

Tip

Not sure which to choose? Start with cloud-hosted endpoints to evaluate the blueprint, then migrate to self-hosted when you need production-grade privacy, throughput, or cost optimization.

Alternative Vector Databases

KDB-X with cuVS (CAGRA) is the default — see the KDB-X configuration guide (all KDBX_* env vars) and the step-by-step EKS + KDB-X cuVS setup runbook. Note a bare KDB-X install needs prep (build the kdbx images, supply a license/bearer, GPU node). To use a different vector database, see:

• KDB.AI – GPU cuVS vector store with public images (works out of the box). Select it with -f deploy/EKS/rag-values-kdbai.yaml.
• Elasticsearch – Hybrid search with BM25 and vector similarity

Refer to the full documentation to learn about the following:

• Minimum Requirements
• Deployment Options
• Configuration Settings
• Common Customizations
• Available Notebooks
• Troubleshooting
• Additional Resources

Utility Scripts

The scripts/ directory contains utilities for testing and interacting with the RAG system:

• SEC Filings Downloader – Download 10-K filings from SEC EDGAR for testing with real financial documents
• Batch Ingestion CLI – Bulk upload datasets with progress tracking
• Retriever API CLI – Query the RAG server from the command line

See scripts/README.md for usage details.

Blog Posts

• NVIDIA NeMo Retriever Delivers Accurate Multimodal PDF Data Extraction 15x Faster
• Finding the Best Chunking Strategy for Accurate AI Responses

Inviting the community to contribute

We're posting these examples on GitHub to support the NVIDIA LLM community and facilitate feedback. We invite contributions! To open a GitHub issue or pull request, see the contributing guidelines.

License

This NVIDIA AI BLUEPRINT is licensed under the Apache License, Version 2.0. This project will download and install additional third-party open source software projects and containers. Review the license terms of these open source projects before use.

Use of the models in this blueprint is governed by the NVIDIA AI Foundation Models Community License.

KX / KDB.AI License

This fork uses a KX vector database — KDB-X by default, or KDB.AI — both of which require a valid KX license. Use of KDB.AI specifically requires:

• KDB.AI License: A valid license from KX. Free trial licenses are available for evaluation (valid for 90 days).
• KDB.AI Terms of Service: Use of KDB.AI is governed by the KX Terms of Service.
• KDB.AI Documentation: KDB.AI Server Setup Guide

To obtain a KDB.AI license, visit kx.com and sign up for access.

KDB-X users bring their own kdb+ license from KX. The open-source adapter has no separate license requirement.

Terms of Use

This blueprint is governed by the NVIDIA Agreements | Enterprise Software | NVIDIA Software License Agreement and the NVIDIA Agreements | Enterprise Software | Product Specific Terms for AI Product. The models are governed by the NVIDIA Agreements | Enterprise Software | NVIDIA Community Model License and the NVIDIA RAG dataset which is governed by the NVIDIA Asset License Agreement. The following models that are built with Llama are governed by the Llama 3.2 Community License Agreement: nvidia/llama-nemotron-embed-1b-v2 and nvidia/llama-nemotron-rerank-1b-v2 and llama-nemotron-embed-vl-1b-v2.

KX / KDB.AI Terms

Use of KDB.AI as the vector database is subject to:

• KX Terms of Service
• KX Privacy Policy
• KDB.AI License Agreement (provided upon registration)

Additional Information

The Llama 3.1 Community License Agreement for the nemotron-nano-12b-v2-vl, llama-3.1-nemoguard-8b-content-safety and llama-3.1-nemoguard-8b-topic-control models. The Llama 3.2 Community License Agreement for the nvidia/llama-nemotron-embed-1b-v2, nvidia/llama-nemotron-rerank-1b-v2 and llama-nemotron-embed-vl-1b-v2 models. The Llama 3.3 Community License Agreement for the llama-3.3-nemotron-super-49b-v1.5 models. Built with Llama. Apache 2.0 for NVIDIA Ingest and for the nemoretriever-page-elements-v2, nemoretriever-table-structure-v1, nemoretriever-graphic-elements-v1, paddleocr and nemoretriever-ocr-v1 models.