Rag Of All Trades

This project is a complete RAG (Retrieval-Augmented Generation) microservice. The service is based on FastAPI, PGVector, Redis, Celery, and LlamaIndex.

The service is built with extensibility in mind and provides a flexible configuration that allows you to easily connect to an arbitrary number of data sources with pre-defined ingestion schedules.

✨ Features

Ingestion from S3 buckets with Everything-to-Markdown conversion via MarkItDown
Ingestion from local directories via LlamaIndex SimpleDirectoryReader
Ingestion from MediaWiki with Wiki-to-Markdown conversion via html2text
SerpAPI ingestion from Google Search results with customizable queries
Jira ingestion from Cloud and on-premise instances via JQL queries, with optional comment loading
Slack ingestion from channels by ID or name/regex pattern, with thread reply support
GitHub ingestion from repository files and issues via Personal Access Token or GitHub App
Notion ingestion from pages and databases via a Notion integration token
Flexible configuration supporting an arbitrary number of connectors
Built with extensibility in mind, allowing for custom connectors with ease

🌐 Connectors included

S3
Directory
MediaWiki
SerpAPI
Jira
Web
Pipedrive
Slack
GitHub
IMAP
OneDrive (OneDrive for Business — App authentication)
Notion

Embeddings support

Local (running arbitrary embedding models from HuggingFace)
OpenRouter
OpenAI

Inference support

OpenRouter
OpenAI

Tech Stack

FastAPI
Vector search with pgvector
Celery-based ingestion pipeline
OpenAI/OpenRouter support for inference and embeddings
Local LLM support for inference and embeddings
LlamaIndex-powered RAG Query Engine
Docker Compose for deployment

⚡️Quick Start

Create a .env file based on the .env.example file.
- The defaults are good enough, you just need to put your OpenRouter key into OPENROUTER_API_KEY
If you use OpenAI or a different OpenAI-compatible endpoint, also update the OPENROUTER_API_BASE variable
By default, a single S3 connector is configured; specify your S3 bucket credentials in the S3_ACCOUNT1_ variables
Create a config.yaml file based on the config.yaml.example file
- The defaults are good enough with openai/gpt-oss-120b:free used for inference and sentence-transformers/all-mpnet-base-v2 for embeddings.
- If you would like to use different models, update the embedding and inference sections accordingly.
Run docker compose up -d --build to start the service
Access the API at :8000
Access the API docs at :8000/docs

API Authentication

By default, the API endpoints require no authentication.

To enable Bearer token authorization, set API_KEY in your .env file:

API_KEY=your-strong-api-key

When set, all requests to the API must include the key as a Bearer token:

Authorization: Bearer <API_KEY>

Requests without the header or with a wrong token will receive a 401 Unauthorized response.

MCP endpoint

The service includes an optional MCP (Model Context Protocol) server at /mcp/ (trailing slash required). It is disabled by default and can be enabled via environment variables.

Enabling MCP

Set the following in your .env file:

MCP_ENABLE=1
MCP_API_KEY=your-strong-api-key

Authentication

All MCP requests require a Bearer token in the Authorization header:

Authorization: Bearer <MCP_API_KEY>

Transport

The MCP server uses stateless HTTP mode (Streamable HTTP transport), so no mcp-session-id header is required. The endpoint accepts JSON-RPC requests.

Available tools

retrieve_chunks - top-k retrieval from vector store with optional metadata filters
rephrase_chunks - LLM-based answer generation over top-k retrieved chunks (requires inference to be configured)

Testing with MCP Inspector

You can use the MCP Inspector to test the MCP endpoint:

./mcp_inspector.sh

This starts the inspector in Docker and prints a URL with pre-filled connection settings.

Connectors

The service supports multiple data sources, including multiple data sources of the same type, each with its own ingestion schedule. The connectors to enable are defined via config.yaml, and their secrets are defined in the .env file.

S3 Connector

The S3 connector ingests documents from S3 buckets and converts them to Markdown format. The connector has the following configuration options:

# config.yaml

sources:
  - type: "s3" # must be s3
    name: "account1" # arbitrary name for the connector, will be stored in metadata
    config:
      endpoint: "${S3_ACCOUNT1_ENDPOINT}" # s3 endpoint
      access_key: "${S3_ACCOUNT1_ACCESS_KEY}" # s3 access key
      secret_key: "${S3_ACCOUNT1_SECRET_KEY}" # s3 secret key
      region: "${S3_ACCOUNT1_REGION}" # s3 region
      use_ssl: "${S3_ACCOUNT1_USE_SSL}" # use ssl for s3 connection, can be True or False
      buckets: "${S3_ACCOUNT1_BUCKETS}" # single entry or comma-separated list i.e. bucket1,bucket2
      schedules: "${S3_ACCOUNT1_SCHEDULES}" # single entry or comma-separated list i.e. 3600,60

  - type: "s3"
    name: "account2"
    config:
      ...

  - type: "s3"
    name: "account3"
    config:
      ...

# .env

S3_ACCOUNT1_ENDPOINT=https://s3.amazonaws.com
S3_ACCOUNT1_ACCESS_KEY=xxx
S3_ACCOUNT1_SECRET_KEY=xxx
S3_ACCOUNT1_REGION=us-east-1
S3_ACCOUNT1_USE_SSL=True
S3_ACCOUNT1_BUCKETS=bucket1,bucket2
S3_ACCOUNT1_SCHEDULES=3600,60

Directory Connector

The directory connector ingests files from a local filesystem directory using LlamaIndex SimpleDirectoryReader. The connector has the following configuration options:

# config.yaml

sources:
  - type: "directory"
    name: "local_docs"
    config:
      path: "/data/docs" # required path to directory
      recursive: true # optional, default true
      required_exts: "txt,md,pdf" # optional, comma-separated extensions
      exclude_hidden: true # optional, default true
      exclude_empty: false # optional, default false
      num_files_limit: 1000 # optional, positive integer
      schedules: "3600"

MediaWiki Connector

The MediaWiki connector ingests documents from MediaWiki sites and converts them to Markdown format. The connector has the following configuration options:

Set load_semantics: true on a wiki that has Semantic MediaWiki installed to attach each page's semantic properties as document metadata, each under a smw_-prefixed key. The property name is lowercased and spaces are replaced with underscores (e.g. Assigned editor becomes smw_assigned_editor) to avoid colliding with the connector's own metadata fields and to give metadata filters a predictable name to construct. System properties (_ASK, _INST, _SKEY, etc.) and subobjects are excluded; a single-valued property is stored as a plain value, and a multi-valued property is stored as a list, filterable via the /api/v1/query metadata filter API's CONTAINS, IN, and NIN operators.

# config.yaml

sources:
  - type: "mediawiki"
    name: "wiki1"
    config:
      # Either api_url OR host (+ optional path/scheme)
      # host: "${MEDIAWIKI1_HOST}"
      # path: "/w/"
      # scheme: "https"
      api_url: "${MEDIAWIKI1_API_URL}"
      page_limit: 500
      namespaces: "0"
      filter_redirects: true
      username: "${MEDIAWIKI1_USERNAME}"  # optional, private wikis
      password: "${MEDIAWIKI1_PASSWORD}"
      # Network overrides (optional) — reverse-proxy bypass / custom TLS
      # verify_ssl: true              # set false to skip TLS cert verification
      # resolve_to_ip: "10.0.0.1"     # connect to this IP, keep Host/SNI as hostname
      # user_agent: "MyBot/1.0"       # override HTTP User-Agent
      # custom_headers:               # extra headers on every API request
      #   Authorization: "Bearer token"
      request_delay: 0.1
      load_semantics: false # optional, query Semantic MediaWiki properties per page (default: false)
      schedules: "${MEDIAWIKI1_SCHEDULES}"

# .env

MEDIAWIKI1_API_URL=https://en.wikipedia.org/w/api.php
MEDIAWIKI1_SCHEDULES=3600

Optional network settings:

Key	Default	Purpose
`verify_ssl`	`true`	TLS certificate verification
`resolve_to_ip`	unset	Connect to this IP while keeping hostname for Host/SNI (like `curl --resolve`)
`user_agent`	mwclient default	Override HTTP `User-Agent` (wins over `custom_headers`)
`custom_headers`	unset	Extra HTTP headers on all MediaWiki API requests

SerpAPI Connector

The SerpAPI connector ingests documents from Google Search results and converts them to Markdown format. The connector has the following configuration options:

# config.yaml

sources:
  - type: "serpapi"
    name: "serp_ingestion1"
    config:
      api_key: "${SERPAPI1_KEY}"
      queries: "${SERPAPI1_QUERIES}"
      schedules: "${SERPAPI1_SCHEDULES}"

  - type: "serpapi"
    name: "serp_ingestion2"
    config:

  - type: "serpapi"
    name: "serp_ingestion3"
    config:

# .env

SERPAPI1_KEY=xxxx
SERPAPI1_QUERIES=aaa
SERPAPI1_SCHEDULES=3600

Web Connector

The Web connector ingests content from web pages using the LlamaIndex BeautifulSoupWebReader (URLs mode) or SitemapReader (sitemap mode). The two modes are mutually exclusive.

URLs mode — scrape a fixed list of pages:

- type: web
  name: web1
  config:
    urls:
      - https://example.com/page1
      - https://example.com/page2
    html_to_text: true   # optional, default true
    schedules: "${WEB1_SCHEDULES}"

Sitemap mode — discover and scrape URLs from a sitemap.xml:

- type: web
  name: web2
  config:
    sitemap_url: https://example.com/sitemap.xml
    include_prefix: "/wiki/"   # optional: only ingest URLs containing this string
    html_to_text: true         # optional, default true
    schedules: "${WEB2_SCHEDULES}"

Note: exclude_prefix and sitemap index (<sitemapindex>) are not supported in this iteration — the underlying SitemapReader only supports include-style filtering and flat sitemaps.

.env variables:

WEB1_SCHEDULES=60
WEB2_SCHEDULES=60

No other credentials are required for public web pages.

Jira Connector

The Jira connector ingests issues from Jira Cloud or on-premise (Server/Data Center) instances using a JQL query. Issue content (summary + description) is converted to Markdown. Metadata collected per issue includes: id, title, url, status, assignee, reporter, labels, project, priority, issue type

Supports two authentication modes:

Basic auth (auth_type: basic) — email + API token, for Jira Cloud
Personal Access Token (auth_type: token) — PAT as Bearer header, for Jira Server / Data Center

# config.yaml

sources:
  - type: "jira"
    name: "jira1"
    config:
      server_url: "${JIRA1_SERVER_URL}"
      auth_type: "basic"              # "basic" or "token"
      email: "${JIRA1_EMAIL}"         # required for auth_type=basic
      api_token: "${JIRA1_API_TOKEN}"
      jql: "${JIRA1_JQL}"
      max_results: 50                 # optional, default 50
      schedules: "${JIRA1_SCHEDULES}"
      # Optional: load top N comments per issue
      load_comments: false            # optional, default false
      max_comments: 10                # optional, default 10

# .env

# Jira Cloud (basic auth)
JIRA1_SERVER_URL=https://your-org.atlassian.net
JIRA1_EMAIL=your-email@example.com
JIRA1_API_TOKEN=your-api-token
JIRA1_JQL=project = MYPROJECT ORDER BY updated DESC
JIRA1_SCHEDULES=3600

# Jira Server / Data Center (Personal Access Token)
# JIRA1_SERVER_URL=https://jira.your-company.com
# JIRA1_API_TOKEN=your-personal-access-token
# (set auth_type: "token" in config.yaml; email is not needed)

Pipedrive Connector

The Pipedrive connector ingests CRM records from Pipedrive using the REST API v1. Supports activities, deals, notes, organizations, persons, products, projects, leads, tasks, and mails. Metadata collected per record includes type-specific fields such as linked entities, pipeline/stage names, assignees, authors, and timestamps.

# config.yaml

sources:
  - type: "pipedrive"
    name: "pipedrive1"
    config:
      api_token: "${PIPEDRIVE1_API_TOKEN}"
      schedules: "${PIPEDRIVE1_SCHEDULES}"
      # Optional: which entity types to load (default: all)
      # load_types:
      #   - activities
      #   - deals
      #   - notes
      #   - organizations
      #   - persons
      #   - products
      #   - projects
      #   - leads
      #   - tasks
      #   - mails
      # max_items: 500          # optional, global per-entity limit (default: unlimited)
      # request_delay: 1        # optional, seconds between API requests (default: 0)
      # max_retries: 3          # optional, retries on failure (default: 1)
      # filter_deals_updated_since: "2025-01-01"
      # filter_activities_updated_since: "2025-01-01"
      # filter_deals_stages_ids:
      #   - 1
      # filter_deals_filter_id: "xxx"
      # filter_organizations_filter_id: "xxx"
      # filter_persons_filter_id: "xxx"
      # filter_mail_folders:
      #   - inbox

# .env

PIPEDRIVE1_API_TOKEN=your-pipedrive-api-token
PIPEDRIVE1_SCHEDULES=3600

SharePoint Connector

The SharePoint connector ingests files from SharePoint document libraries or site pages using the LlamaIndex SharePoint reader. Authentication is via Microsoft Entra ID (client credentials).

Supports two modes:

File mode (sharepoint_type: file, default): load files from a drive/folder
Page mode (sharepoint_type: page): load SharePoint site pages

# config.yaml

sources:
  # Loading files from a SharePoint drive
  - type: "sharepoint"
    name: "sharepoint1"
    config:
      client_id: "${SHAREPOINT1_CLIENT_ID}"
      client_secret: "${SHAREPOINT1_CLIENT_SECRET}"
      tenant_id: "${SHAREPOINT1_TENANT_ID}"
      # sharepoint_site_id can be provided instead of sharepoint_site_name
      sharepoint_site_name: "MySite"
      # sharepoint_host_name and sharepoint_relative_url are passed through to
      # the reader but do NOT replace sharepoint_site_name / sharepoint_site_id
      # for site lookup. At least one of site_name or site_id is required.
      # sharepoint_host_name: "contoso.sharepoint.com"
      # sharepoint_relative_url: "sites/YourSiteName"
      # sharepoint_folder_id can be provided instead of sharepoint_folder_path
      sharepoint_folder_path: "Documents/Reports"
      sharepoint_type: "file"                   # "file" (default) or "page"
      recursive: true
      schedules: "${SHAREPOINT1_SCHEDULES}"

  # Loading SharePoint site pages
  - type: "sharepoint"
    name: "sharepoint2"
    config:
      client_id: "${SHAREPOINT2_CLIENT_ID}"
      client_secret: "${SHAREPOINT2_CLIENT_SECRET}"
      tenant_id: "${SHAREPOINT2_TENANT_ID}"
      sharepoint_site_name: "TeamSite"
      sharepoint_type: "page"
      schedules: "${SHAREPOINT2_SCHEDULES}"

# .env

SHAREPOINT1_CLIENT_ID=your-azure-app-client-id
SHAREPOINT1_CLIENT_SECRET=your-azure-app-client-secret
SHAREPOINT1_TENANT_ID=your-azure-tenant-id
SHAREPOINT1_SCHEDULES=3600

SHAREPOINT2_CLIENT_ID=your-azure-app-client-id
SHAREPOINT2_CLIENT_SECRET=your-azure-app-client-secret
SHAREPOINT2_TENANT_ID=your-azure-tenant-id
SHAREPOINT2_SCHEDULES=3600

Slack Connector

The Slack connector ingests messages from Slack channels. Each message and each thread reply is ingested as an individual item. Channels can be specified by ID or resolved via name/regex patterns.

Authentication requires a Slack bot token with the following scopes: channels:history, groups:history, channels:read, groups:read, users:read. Invite the bot to each target channel (/invite @YourBot).

# config.yaml

sources:
  - type: "slack"
    name: "slack1"
    config:
      token: "${SLACK1_TOKEN}"
      channel_ids: "${SLACK1_CHANNEL_IDS}"           # comma-separated channel IDs
      schedules: "${SLACK1_SCHEDULES}"

  # Using channel name patterns instead of IDs:
  - type: "slack"
    name: "slack2"
    config:
      token: "${SLACK2_TOKEN}"
      channel_patterns: "${SLACK2_CHANNEL_PATTERNS}" # comma-separated names or regex, e.g. "general,^dev.*"
      channel_types: "public_channel,private_channel" # optional, default public_channel,private_channel
      earliest_date: "2024-01-01"                    # optional
      latest_date: "2025-01-01"                      # optional, requires earliest_date
      schedules: "${SLACK2_SCHEDULES}"

# .env

SLACK1_TOKEN=xoxb-your-bot-token
SLACK1_CHANNEL_IDS=C0123456789,C9876543210
SLACK1_CHANNEL_PATTERNS=general,^dev.*
SLACK1_SCHEDULES=3600
SLACK2_TOKEN=xoxb-your-second-bot-token
SLACK2_CHANNEL_PATTERNS=^marketing.*,^sales.*
SLACK2_SCHEDULES=3600

channel_ids and channel_patterns are mutually exclusive. latest_date requires earliest_date.

IMAP Connector

The IMAP connector ingests emails from any IMAP-capable mail server (Gmail, Outlook, self-hosted, etc.) via implicit TLS (IMAP4_SSL) or STARTTLS. Each email becomes a document with the subject as the title and the parsed body as content. Metadata collected per email includes: subject, from, to, cc, bcc, date, mailbox, message_id.

Deduplication is based on the Message-ID header (cross-mailbox safe). Falls back to mailbox+UID when Message-ID is absent.

Server certificates are always verified (hostname + trust chain) — self-signed certificates will be rejected unless the server's CA is trusted by the environment running the connector.

# config.yaml

sources:
  - type: "imap"
    name: "imap1"
    config:
      host: "${IMAP1_HOST}"
      port: 993                     # optional, default 993 (IMAPS), or 143 when use_starttls is set
      username: "${IMAP1_USERNAME}"
      password: "${IMAP1_PASSWORD}" # app-specific password for Gmail
      mailboxes: "${IMAP1_MAILBOXES}" # optional, comma-separated; remove this line (not just the env var) to ingest all mailboxes
      since: "2024-01-01"            # optional, only ingest messages on or after this date (YYYY-MM-DD)
      use_starttls: false            # optional, default false; connect plaintext then upgrade via STARTTLS instead of implicit TLS
      schedules: "${IMAP1_SCHEDULES}"
      #request_delay: 0  # optional, seconds between items; raise for rate-limited providers (e.g. Gmail)

# .env

IMAP1_HOST=imap.gmail.com
IMAP1_USERNAME=your-email@gmail.com
IMAP1_PASSWORD=your-app-specific-password
IMAP1_MAILBOXES=INBOX,Sent
IMAP1_SCHEDULES=3600

SHAREPOINT1_CLIENT_ID=your-azure-app-client-id
SHAREPOINT1_CLIENT_SECRET=your-azure-app-client-secret
SHAREPOINT1_TENANT_ID=your-azure-tenant-id
SHAREPOINT1_SCHEDULES=3600

OneDrive Connector

The OneDrive connector ingests files from Microsoft OneDrive for Business (Microsoft 365) using App authentication (client credentials). Files can be selected by folder ID, folder path, file IDs, or file paths. Recursive subfolder traversal and MIME type filtering are supported.

Note: Only OneDrive for Business is supported. OneDrive Personal accounts are not supported.

The Azure app registration needs the application permission Files.Read.All with admin consent granted. Without it, Graph API calls fail with a 403 error.

# config.yaml

sources:
  - type: "onedrive"
    name: "onedrive1"
    config:
      client_id: "${ONEDRIVE1_CLIENT_ID}"
      client_secret: "${ONEDRIVE1_CLIENT_SECRET}"
      tenant_id: "${ONEDRIVE1_TENANT_ID}"
      userprincipalname: "${ONEDRIVE1_USER_PRINCIPAL_NAME}"
      folder_path: "Documents/Reports"  # optional: hardcode directly in config
      folder_id:                        # optional: OneDrive folder ID
      file_ids:                         # optional: comma-separated file IDs
      file_paths:                       # optional: comma-separated file paths
      mime_types:                       # optional: comma-separated MIME types to filter
      recursive: true                   # optional, default true
      max_file_size_mb: 50              # optional, default 50; files larger than this are skipped
      schedules: "${ONEDRIVE1_SCHEDULES}"

# .env

ONEDRIVE1_CLIENT_ID=your-azure-app-client-id
ONEDRIVE1_CLIENT_SECRET=your-azure-app-client-secret
ONEDRIVE1_TENANT_ID=your-azure-tenant-id
ONEDRIVE1_USER_PRINCIPAL_NAME=user@your-org.onmicrosoft.com
ONEDRIVE1_SCHEDULES=3600

GitHub Connector

The GitHub connector ingests repository files and optionally issues from a GitHub repository. Files can be filtered by extension and directory. Authentication supports a Personal Access Token or GitHub App credentials. The 1 suffix in variable names (e.g. GITHUB1_*) supports multiple GitHub connector instances.

# config.yaml

sources:
  - type: "github"
    name: "github1"
    config:
      personal_token: "${GITHUB1_PERSONAL_TOKEN}"
      # GitHub App auth (mutually exclusive with personal_token):
      #github_app_id: "${GITHUB1_APP_ID}"
      #github_app_installation_id: "${GITHUB1_APP_INSTALLATION_ID}"
      #github_app_private_key: "${GITHUB1_APP_PRIVATE_KEY}"
      owner: "${GITHUB1_OWNER}"
      repo: "${GITHUB1_REPO}"
      branch: "main"                    # optional, default "main" (mutually exclusive with commit_sha)
      #commit_sha: ""                   # optional (mutually exclusive with branch)
      include_extensions: "md,py"       # optional, comma-separated (mutually exclusive with exclude_extensions)
      #exclude_extensions: ""           # optional (mutually exclusive with include_extensions)
      #include_directories: ""          # optional, comma-separated (mutually exclusive with exclude_directories)
      #exclude_directories: ""          # optional (mutually exclusive with include_directories)
      include_issues: false             # optional, default false
      #include_issues_labels: ""        # optional, comma-separated (mutually exclusive with exclude_issues_labels)
      #exclude_issues_labels: ""        # optional (mutually exclusive with include_issues_labels)
      concurrent_requests: 5            # optional, default 5
      schedules: "${GITHUB1_SCHEDULES}"

# .env

GITHUB1_PERSONAL_TOKEN=your-personal-access-token
# GitHub App auth (alternative to personal token):
#GITHUB1_APP_ID=
#GITHUB1_APP_INSTALLATION_ID=
#GITHUB1_APP_PRIVATE_KEY=
GITHUB1_OWNER=your-org-or-username
GITHUB1_REPO=your-repo-name
GITHUB1_SCHEDULES=3600

Notion Connector

The Notion connector ingests pages and database entries from a Notion workspace using a Notion integration token. Pages can be selected explicitly by ID, by database, or all accessible pages are ingested when neither is specified.

# config.yaml

sources:
  - type: "notion"
    name: "notion1"
    config:
      integration_token: "${NOTION1_INTEGRATION_TOKEN}"
      page_ids: "${NOTION1_PAGE_IDS}"          # optional: comma-separated page IDs
      database_ids: "${NOTION1_DATABASE_IDS}"  # optional: comma-separated database IDs
      request_delay: 0.3                       # optional: delay between API calls in seconds
      schedules: "${NOTION1_SCHEDULES}"

# .env

NOTION1_INTEGRATION_TOKEN=secret_your-notion-integration-token
NOTION1_PAGE_IDS=page-id-1,page-id-2
NOTION1_DATABASE_IDS=database-id-1
NOTION1_SCHEDULES=3600

Reference of the `config.yaml`

The config.yaml file contains the main configuration of the service.

Common connector parameters

The following parameters are supported by all connector types:

Parameter	Type	Default	Description
`enabled`	bool	`true`	Set to `false` to skip this connector entirely — no Celery task or Beat schedule is registered.
`schedules`	string	—	Cron expression or interval (in seconds) defining how often the connector runs.
`request_delay`	float	`0`	Delay in seconds between processing each item. Useful for rate-limiting requests to external APIs.

Environment variables (${...}) in the config file are evaluated at runtime.

sources: # holds the list of sources to ingest from (Connectors)

  - type: # type of the connector (s3, directory, mediawiki, serpapi, jira, etc.)
    name: # arbitrary name for the connector, will be stored in metadata
    enabled: true  # optional; set to false to skip this connector entirely
    config:
      # connector specific configuration
      schedules: "${S3_ACCOUNT1_SCHEDULES}"
      request_delay: 0  # optional, delay in seconds between items (default: 0)

# configures models and dimensions for embeddings
embedding:
  provider: openrouter # `openrouter`/`openai` or `local` for local HuggingFace embeddings
  model_config: text-embedding-3-small # model to use
  embedding_dim: 1536 # dimensions (check with the model docs)

# configures the LLM provider and model
inference:
  provider: openrouter # `openrouter`/`openai`
  model_config: gpt-4o # model to use

# vector store configuration
vector_store:
  table_name: embeddings
  hybrid_search: true # whether to use hybrid search or not
  chunk_size: 512 # chunk size for vector indexing
  chunk_overlap: 50 # overlap between chunks
  # hnsw indexes settings
  hnsw:
    hnsw_m: 16 # number of neighbors
    hnsw_ef_construction: 64 # ef construction parameter for HNSW
    hnsw_ef_search: 40 # ef search parameter for HNSW
    hnsw_dist_method: vector_cosine_ops # distance metric for HNSW

Embeddings and Inference configuration examples

Embeddings-only HuggingFace local model

You can configure the service to use local embeddings only, in this mode you can use any embedding model supported by HuggingFace. Inference is disabled in this mode, so you won't be able to use the rephrase endpoint.

# config.yaml

embedding:
  provider: local
  # you can use any embedding model supported by HuggingFace
  model_config: sentence-transformers/all-MiniLM-L6-v2
  embedding_dim: 384

inference:
  provider: None
  model_config: None

Embeddings-only OpenRouter/OpenAI model

You can configure the service to use remote embeddings, in this mode you can use any embedding model supported by OpenRouter/OpenAI. Inference is disabled in this mode, so you won't be able to use the rephrase endpoint.

# config.yaml

embedding:
  provider: openrouter
  model_config: text-embedding-3-small
  embedding_dim: 1536

inference:
  provider: None
  model_config: None

You must set OPENROUTER_API_KEY and OPENROUTER_API_BASE in the .env file.

Embeddings and inference OpenRouter/OpenAI model

You can configure the service to use remote embeddings and remote inference, in this mode you can use any embedding and inference models supported by OpenRouter/OpenAI.

# config.yaml

embedding:
  provider: openrouter
  model_config: text-embedding-3-small
  embedding_dim: 1536

inference:
  provider: openrouter
  model_config: gpt-4o

You must set OPENROUTER_API_KEY and OPENROUTER_API_BASE in the .env file.

API

The following API endpoints are available:

/api/v1/query

This endpoint is used to perform a query against the vector store:

curl -X 'POST' \
  'http://localhost:8000/api/v1/query' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "AWS Services",
  "top_k": 5
}'

If API_KEY is configured, include the Authorization header:

curl -X 'POST' \
  'http://localhost:8000/api/v1/query' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "AWS Services",
  "top_k": 5
}'

Response example:

{
  "references": [
    {
      "source_name": null,
      "source_type": null,
      "url": null,
      "score": 0.6172290216224814,
      "title": null,
      "text": "You can also\n\nrequire WAF Captcha challenges for suspicious...",
      "extras": {
        "source": "s3",
        "key": "aws-overview.pdf",
        "checksum": "5b4da9267b0b861792d1163fcc9f0550",
        "version": 1,
        "format": "markdown"
      }
    },
    {...},
    {...}
  ],
  "raw": [
    "Score: 0.6172 | Text: You can also\n\nrequire WAF Captcha challenges for suspicious...",
    "Score: 0.5172 | Text: You can also\n\nrequire WAF Captcha challenges for suspicious...",
    "Score: 0.3172 | Text: You can also\n\nrequire WAF Captcha challenges for suspicious..."
  ]
}

/api/v1/rephrase

This endpoint rephrases the query and provides the best answer.

This endpoint requires inference to be configured in the config.yaml.

curl -X 'POST' \
  'http://localhost:8000/api/v1/rephrase' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "WAF Captcha challenges for suspicious requests"
}'

If API_KEY is configured, include the Authorization header:

curl -X 'POST' \
  'http://localhost:8000/api/v1/rephrase' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "WAF Captcha challenges for suspicious requests"
}'

Response format:

{
  "answer": "You can configure AWS WAF to require Captcha challenges for suspicious requests based on:\n- Request rate and attributes",
  "references": [
    {
      "source_name": null,
      "source_type": null,
      "url": null,
      "score": 0.5415070280718167,
      "title": null,
      "extras": {
        "source": "s3",
        "key": "aws-overview.pdf",
        "checksum": "5b4da9267b0b861792d1163fcc9f0550",
        "version": 1,
        "format": "markdown"
      }
    }
  ]
}

/health

This endpoint checks the health of the service.

curl -X 'GET' \
  'http://localhost:8000/health' \
  -H 'accept: application/json'

Response example:

{
  "status": "ok",
  "vector_store_loaded": true,
  "celery_healthy": true
}

Integration examples

OpenWebUI

TODO

LibreChat

TODO

LobeHub

TODO

Anything-LLM

TODO

🔧 Development

Pre-commit hooks

This project uses prek (a fast, drop-in alternative to pre-commit) to enforce formatting and linting on every commit.

Install prek (once, globally):

# Using pip
pip install prek

# Or using the standalone installer (Linux/macOS)
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/j178/prek/releases/latest/download/prek-installer.sh | sh

Install the hooks (once, per clone):

prek install

From that point on, every git commit will automatically run:

Hook	What it does
`trailing-whitespace`	Removes trailing whitespace
`end-of-file-fixer`	Ensures files end with a newline
`check-yaml`	Validates YAML syntax
`check-merge-conflict`	Detects unresolved merge conflict markers
`ruff` (lint)	Lints Python with auto-fix (pycodestyle, pyflakes, isort, pyupgrade)
`ruff-format`	Formats Python code (replaces black)

Run hooks manually (without committing):

prek run --all-files

Ruff configuration is in pyproject.toml under [tool.ruff].

🔩 Maintenance

Wipe ingested records

Use scripts/wipe_ingested.py to selectively delete ingested records from the database.

Run it inside the running api container:

docker compose exec api python scripts/wipe_ingested.py --all                                        # full wipe
docker compose exec api python scripts/wipe_ingested.py --source pipedrive1                           # by source_name
docker compose exec api python scripts/wipe_ingested.py --source pipedrive1 --filter entity_type=note

--filter accepts a single key=value pair matched against the connector metadata. Requires --source.

✨ Contributions

Contributions, suggestions, bug reports, and fixes are welcome!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Rag Of All Trades

✨ Features

🌐 Connectors included

Embeddings support

Inference support

Tech Stack

⚡️Quick Start

API Authentication

MCP endpoint

Enabling MCP

Authentication

Transport

Available tools

Testing with MCP Inspector

Connectors

S3 Connector

Directory Connector

MediaWiki Connector

SerpAPI Connector

Web Connector

Jira Connector

Pipedrive Connector

SharePoint Connector

Slack Connector

IMAP Connector

OneDrive Connector

GitHub Connector

Notion Connector

Reference of the config.yaml

Common connector parameters

Embeddings and Inference configuration examples

Embeddings-only HuggingFace local model

Embeddings-only OpenRouter/OpenAI model

Embeddings and inference OpenRouter/OpenAI model

API

/api/v1/query

/api/v1/rephrase

/health

Integration examples

OpenWebUI

LibreChat

LobeHub

Anything-LLM

🔧 Development

Pre-commit hooks

🔩 Maintenance

Wipe ingested records

✨ Contributions

Star History

Reference of the `config.yaml`