Monolith API with vLLM/Ollama, PostgreSQL (pgvector), MCP, PDF RAG, React, and Nginx

This repository implements a monolithic chat application with:

• FastAPI backend (monolith API)
• PostgreSQL + pgvector for chats, messages, and document chunk embeddings
• vLLM or Ollama for LLM inference (OpenAI-compatible API); default Docker setup uses Ollama with Meta Llama or other models
• Token-aware context handling with a sliding window and basic compaction
• MCP tools served via an MCP server and proxied through MCPO, including PinchTab browser automation for “browse the web” in chat
• Chat-with-PDF (RAG) using document chunk embeddings stored in pgvector
• React (Vite + TypeScript) frontend with chat naming and optional browse (PinchTab) toggle
• Nginx as a single entrypoint reverse proxy
• Docker Compose to orchestrate everything

The architecture follows:

• Monolith API vLLM React plan – backend, DB, vLLM/Ollama, MCP, PDF RAG, React, Nginx
• PinchTab MCP and chat naming plan – PinchTab tools, browse toggle, chat title create/edit, tool-use loop

---

Table of contents

• High-level architecture
• Repository layout and folder structure
• Component overview
• Diagrams
• Core features
• Configuration
• Running locally with Docker Compose
• Running without Docker
• Notes and limitations

---

High-level architecture

• Client: React app in the browser.
• Nginx: Single entrypoint; routes / → frontend, /api/ → FastAPI, /mcp/ → MCPO. Ollama/vLLM is called by the API directly (not via Nginx in the default Docker setup).
• Backend: FastAPI talks to PostgreSQL, Ollama/vLLM (internal network), and MCPO (for MCP/PinchTab tools).
• Data: PostgreSQL with pgvector for chats, messages, documents, and chunk embeddings.

---

Repository layout and folder structure

vllm-chatgpt/
├── .cursor/
│   └── plans/
│       ├── monolith_api_vllm_react_d0097a35.plan.md
│       └── pinchtab_mcp_and_chat_naming_f318797b.plan.md
├── .env.example
├── .gitignore
├── README.md
├── docker-compose.yml
├── backend/
│   ├── __init__.py
│   ├── config.py              # Settings (DB, vLLM/Ollama URL, context, embeddings, MCPO)
│   ├── db.py                   # Async SQLAlchemy engine/session, init_db, pgvector
│   ├── main.py                 # FastAPI app, routers, startup init_db
│   ├── models.py               # Chat, Message, Document, DocumentChunk (Vector)
│   ├── schemas.py              # Pydantic: ChatRead, MessageRead, DocumentUpload, etc.
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── routers/
│   │   ├── __init__.py
│   │   ├── chats.py            # Chat CRUD, send message, vLLM + optional RAG/tools
│   │   ├── documents.py        # PDF upload, chunking, embeddings, pgvector
│   │   └── mcp.py              # Proxy POST /api/mcp/tools/{tool_name} → MCPO
│   ├── services/
│   │   ├── __init__.py
│   │   ├── context.py          # Sliding window + compaction under token budget
│   │   ├── embeddings.py       # sentence-transformers for RAG
│   │   └── vllm.py             # Ollama/vLLM client, chat + tool-use loop
│   └── utils/
│       ├── __init__.py
│       └── tokens.py           # Llama tokenizer-based token counting
├── frontend/
│   ├── src/
│   │   ├── api.ts              # Client for /api/chats, documents, sendMessage, updateChat
│   │   ├── App.tsx             # Chat UI, sidebar, PDF upload, browse toggle
│   │   └── main.tsx
│   ├── index.html
│   ├── Dockerfile
│   ├── package.json
│   ├── tsconfig.json
│   └── vite.config.ts
├── mcp_server/
│   ├── __init__.py
│   ├── main.py                 # FastMCP server: PinchTab tools (create_instance, navigate, snapshot, action, text)
│   ├── requirements.txt
│   └── Dockerfile
├── nginx/
│   └── nginx.conf              # Reverse proxy: / → frontend, /api/ → API, /mcp/ → MCPO
└── vllm/                       # Optional local vLLM / venv (not required if using Ollama)

---

Component overview

Component	Purpose
backend	FastAPI monolith: chat/message CRUD, token-aware context, vLLM/Ollama client, PDF RAG (embeddings + pgvector), MCP proxy.
frontend	Vite + React + TypeScript: list/create/rename chats, message list, input, PDF upload, “Enable browsing (PinchTab)” toggle.
mcp_server	FastMCP server exposing PinchTab tools (create instance, navigate, snapshot, action, text); used by MCPO.
nginx	Single entrypoint; proxies / → frontend, /api/ → backend, /mcp/ → MCPO.
docker-compose	Orchestrates postgres (pgvector), ollama, api, pinchtab, mcpo, frontend, nginx.

Detailed component description

• Backend (FastAPI) – Entrypoint is main.py, which mounts routers under /api. config.py holds DB URL, LLM base URL, context window, embedding model, and MCPO URL. db.py provides async SQLAlchemy and init_db (tables + pgvector). routers/chats.py implements chat CRUD, PATCH for title, and send-message with sliding-window context, optional RAG (document_id), and optional tool-use loop when browse is enabled. routers/documents.py handles PDF upload, text extraction, chunking, embeddings, and storage in pgvector. routers/mcp.py proxies tool invocations to MCPO. services/vllm.py calls Ollama/vLLM for chat completions and runs the tool loop (tool_calls → MCPO → append tool result → call LLM again). services/context.py applies sliding window and token-based compaction. services/embeddings.py uses sentence-transformers for RAG. utils/tokens.py counts tokens with the Llama tokenizer.

• Frontend (React) – Vite + TypeScript SPA. api.ts defines the client for chats, messages, document upload, and chat update. App.tsx provides the sidebar (chat list, create, rename), main chat view (messages, input), “Enable browsing (PinchTab)” checkbox, and PDF upload. All requests use relative paths so Nginx can proxy.

• MCP server – FastMCP app in mcp_server/main.py. Exposes PinchTab tools (create instance, navigate, snapshot, action, text) by calling the PinchTab HTTP API (PINCHTAB_BASE_URL). MCPO runs this server (e.g. via stdio) and exposes the same tools over HTTP/OpenAPI for the backend.

• Nginx – Single public entrypoint. Proxies / to the frontend dev server, /api/ to the FastAPI backend (keeping the /api prefix), and /mcp/ to MCPO. Sets client_max_body_size for PDF uploads. Does not proxy to Ollama; the API reaches Ollama on the Docker network.

• Docker Compose – postgres runs pgvector. ollama serves the LLM (OpenAI-compatible). api depends on postgres and mcpo; uses DATABASE_URL, VLLM_BASE_URL, MCPO_BASE_URL, etc. pinchtab runs the PinchTab browser automation server. mcpo runs MCPO with the MCP server (PinchTab tools) and PINCHTAB_BASE_URL. frontend runs the Vite dev server. nginx depends on api, ollama, frontend, mcpo and exposes port 80.

---

Diagrams

1. Request flow (user sends a message)

2. Browse (PinchTab) tool-use loop

When the user enables “Enable browsing (PinchTab)”, the backend sends tools to the LLM and runs a tool-use loop until the model returns a final text reply:

3. PDF RAG flow

---

Core features

Chat and database model

• POST /api/chats – Create a new chat (optional title); returns chat_id.
• PATCH /api/chats/{chat_id} – Update chat title (chat naming).
• POST /api/chats/{chat_id}/messages – Send user message; backend loads recent messages, applies sliding window and context compaction, calls Ollama/vLLM, saves user + assistant messages, returns both.

All messages and chats are stored in PostgreSQL.

Token utility and context compaction

• backend/utils/tokens.py – Uses the Llama tokenizer (e.g. via transformers.AutoTokenizer) to count tokens for text and message lists.
• backend/services/context.py – Keeps all system messages; for others, keeps the last N messages (SLIDING_WINDOW_SIZE, default 5); if over CONTEXT_WINDOW, drops oldest non-system messages until within budget.

PDF RAG with pgvector

• POST /api/documents/upload – Accept PDF; extract text (pypdf); chunk with overlap; embed with sentence-transformers (e.g. all-MiniLM-L6-v2); store in documents and document_chunks with pgvector.
• POST /api/chats/{chat_id}/messages?document_id=... – Embed user query; vector similarity search on document_chunks for that document; inject top-k chunks as context into the system prompt; then normal sliding-window + LLM call.

MCP + MCPO + PinchTab

• mcp_server/main.py – FastMCP server with PinchTab tools: create instance, navigate, snapshot, action, text (calls PinchTab HTTP API via PINCHTAB_BASE_URL).
• mcpo container runs MCPO wrapping the MCP server, exposing tools over HTTP/OpenAPI. pinchtab container runs PinchTab.
• backend/routers/mcp.py – POST /api/mcp/tools/{tool_name} proxies tool calls to MCPO. When “Enable browsing (PinchTab)” is on, the backend sends PinchTab tools to the LLM and runs the tool-use loop (tool_calls → MCPO → result → LLM until done).

Chat naming and browse toggle

• Chat name: Optional title on create; Edit in sidebar to rename (PATCH /api/chats/{chat_id}).
• Browse (PinchTab): Footer checkbox “Enable browsing (PinchTab)” sets use_browse; backend uses a tool-capable model and the tool loop for that request.

React frontend

• Sidebar: chat list, new chat (optional name), edit to rename.
• Main: message history, input, “Enable browsing (PinchTab)” checkbox, PDF upload for RAG.
• All API calls use relative paths so Nginx can proxy (/api/...).

---

Configuration

Copy .env.example to .env and adjust:

cp .env.example .env

Area	Variables	Notes
Database	POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, DATABASE_URL	SQLAlchemy URL for backend
Ollama / vLLM	VLLM_BASE_URL, VLLM_MODEL_NAME, VLLM_MODEL_NAME_TOOLS, CONTEXT_WINDOW, SLIDING_WINDOW_SIZE	Default Ollama: http://ollama:11434/v1; use a tool-capable model for browse (e.g. Llama 3.1, Qwen)
Embeddings	EMBEDDING_MODEL_NAME, EMBEDDING_DIM	e.g. all-MiniLM-L6-v2, 384
MCPO	MCPO_BASE_URL, MCPO_API_KEY	Backend sends Authorization: Bearer <key> to MCPO
PinchTab	PINCHTAB_BASE_URL	In Docker: http://pinchtab:9867; local: http://localhost:9867
Hugging Face	HF_TOKEN	For gated models if using vLLM

---

Running locally with Docker Compose

Prerequisites

• Docker and Docker Compose
• For vLLM: GPU recommended; for Ollama (default): CPU or GPU

Steps

1. Configure env:

   cd vllm-chatgpt
   cp .env.example .env
   # Edit .env (DB password, VLLM_MODEL_NAME, VLLM_MODEL_NAME_TOOLS, HF_TOKEN if needed)

2. Start the stack:

   docker compose up --build

   Brings up: postgres (pgvector), ollama, api, pinchtab, mcpo, frontend, nginx.

3. Use the app:

   • Open http://localhost (Nginx → frontend).
   • Create a chat (optional name), send messages. Use Edit in sidebar to rename.
   • Enable “Enable browsing (PinchTab)” for browser tools (use a tool-capable model in Ollama).
   • Upload a PDF to use RAG; send messages with that chat/document to query the PDF.

4. Stop:

   docker compose down

---

Running backend and frontend without Docker (optional)

Backend

cd backend
python -m venv .venv
source .venv/bin/activate   # or .venv\Scripts\activate on Windows
pip install -r requirements.txt
# Set DATABASE_URL, VLLM_BASE_URL, etc. (e.g. from ../.env)
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Frontend

cd frontend
npm install
npm run dev -- --host 0.0.0.0 --port 5173

Use http://localhost:8000 for API; optionally run Nginx to mirror Docker routing.

---

Notes and limitations

• vLLM: GPU images may not run on all hosts (e.g. some macOS). You can point VLLM_BASE_URL to a remote vLLM instance or use Ollama (default in Docker).
• Browse (PinchTab): Enable only when needed; use a model that supports tool calling (e.g. Llama 3.1, Qwen). Do not expose PinchTab to the public internet.
• Auth: No user auth in this version; chats are not scoped by user.