All 5 prompts assume a vector database (not JSON/SQLite) for session/message storage. Each prompt is designed to generate a detailed HLD diagram in tools like Mermaid, Lucidchart, or draw.io.
Draw a detailed high-level architecture diagram for a real-time AI voice chatbot pipeline built as a self-hosted Vapi replacement.
Components and flows:
Browser / Client Layer:
- User presses mic button → Web Audio API creates AudioContext and AnalyserNode
- VAD (Voice Activity Detection) loop runs at 60fps using FFT 512-point RMS analysis
- On silence ≥ SILENCE_DELAY ms (configurable, ~1800ms after speech detected), MediaRecorder stops and Blob is assembled
- Blob sent via HTTP POST to /api/stt (multipart/form-data, WAV/WebM)
- If no speech within noSpeechMs (4s first round, 4s follow-up), call ends automatically
Speech-to-Text (STT) Layer:
- Flask /api/stt endpoint receives audio blob
- Forwards to Gemini 2.0 Flash (transcribe_audio) via google-generativeai SDK
- Returns JSON {transcript: "..."} to client
Filler Audio (Perceived Latency Reduction):
- Immediately after VAD stops, a random pre-generated filler clip plays ("Hmm let me think...", "So yeah...", "Okay so...", "Right umm...")
- 4 MP3 files stored in /static/filler_1..4.mp3, generated once via ElevenLabs with the same cloned voice
- Filler plays instantly (no network hop) while STT+LLM+TTS runs in background
- Filler audio paused when real TTS response starts playing
LLM Inference Layer:
- Client POSTs transcript to /api/prompt (non-streaming for voice)
- Flask /api/prompt builds system prompt via skills.py build_system_instruction() — checks for _persona.md skill which replaces default prompt entirely
- Gemini Flash generates response (model fallback chain: gemini-3-flash → gemini-2.5-flash)
- Returns JSON {text, session_id}
Text-to-Speech (TTS) Layer:
- Full response POSTed to /api/tts {text: reply}
- /api/tts tries ElevenLabs keys in order: key1+voice1 → key2+voice2 → Gemini TTS fallback
- ElevenLabs: streams MP3 via /v1/text-to-speech/{voice_id}/stream?output_format=mp3_44100_128&optimize_streaming_latency=3
- Each key has its own cloned voice ID (custom male voice)
- model_id: eleven_flash_v2_5 (low-latency)
- Gemini fallback: returns WAV via generativelanguage.googleapis.com/v1beta (generate_tts in gemini.py)
- Client creates Audio element, sets speaking state on audio.onplaying (not before fetch)
Bye Detection:
- After STT, transcript checked against regex: /\b(bye|goodbye|see you|take care|that's all|nothing else)\b/i
- If matched: bot speaks a goodbye message, then ends call and switches to chat widget
- Prevents infinite loop when user wants to leave
State Machine:
- idle → listening (mic open) → thinking (filler plays, STT+LLM running) → speaking (real TTS playing) → listening (4s timeout) → idle
- UI shows "Listening to you..." during both listening and thinking states (seamless feel)
- "Jai's AI Twin is speaking" only when audio.onplaying fires
Session Persistence:
- Each voice turn appended to vector DB as {role:user, content:transcript} and {role:assistant, content:full_response}
- session_id maintained across voice and text modes (shared between voice modal and chat widget)
- Vector DB stores embedding of each message for semantic retrieval in long sessions
Show all network hops, async boundaries, and error/fallback paths clearly.
Draw a detailed high-level architecture diagram for the multi-turn conversation and session management system in a Flask-based AI chat application.
Components and flows:
Session Lifecycle:
- New user lands on page → client checks localStorage for session_id
- If none, POST /api/sessions → creates new session record in vector DB (sessions collection)
- Schema: {session_id, user_id, title, created_at, updated_at, message_count}
- If session_id exists, GET /api/sessions/{id} to validate it still belongs to the user
- Sidebar lists all sessions via GET /api/sessions (paginated, sorted by updated_at desc)
Message Storage in Vector DB:
- Each message stored as a document: {session_id, role, content, timestamp, embedding}
- Embedding generated by text-embedding model at write time
- On each new user message, retrieve top-K semantically similar past messages (cross-session optional)
- get_conversation_history() returns ordered list for LLM context window
Context Window Assembly (prompt_engine.py):
- PersonalityEngine.build_context(session_id, new_message):
1. Fetch last N messages by recency (hard cap for token budget)
2. Optionally inject top-K semantically retrieved messages as "memory" block
3. Prepend system prompt (role variation, personality, date)
4. Return final messages[] array for Gemini API
LLM Call & Response Streaming:
- stream_generate(messages, model) → yields SSE deltas to client
- Client appends deltas to DOM via renderMd() (marked.js + highlight.js)
- On [DONE] signal: full response text assembled, POST /api/sessions/{id}/messages to persist
Session Branching & Management:
- User can rename session (PATCH /api/sessions/{id})
- Delete session: DELETE /api/sessions/{id} → cascades to all messages in vector DB
- "New Chat" button: creates new session, resets client state
- Admin panel: view all users' sessions, message counts, last active
Rate Limiting & Quotas:
- Flask-Limiter per UID: 60 req/min on /api/chat
- is_pro(uid) check: all users currently return True (open access)
- Per-session message cap: enforce_session_limit() archives oldest sessions beyond N
Show data flows, vector DB read/write operations, SSE streaming boundaries, and failure modes.
Draw a detailed high-level architecture diagram for the authentication and multi-tenant access control system.
Components and flows:
Primary Auth — Google OAuth2 (Flask-Dance):
- User clicks "Sign in with Google" → /auth/google → OAuth2 redirect
- Google redirects to /auth/google/authorized with code
- auth.py exchanges code for token, stores in google_bp.token
- _get_user_id() extracts sub (Google user ID) from id_token JWT
- User ID stored in Flask session (server-side, signed cookie)
- Token stored in file: tokens/{user_id}.json for refresh persistence
- Background thread refreshes tokens every hour via load_token() + requests OAuth refresh endpoint
Role & Access Control:
- is_admin(uid): checks uid against admin_emails list (admin_config.json)
- is_pro(uid): returns True for all users (open-source, full access)
- @login_required decorator: validates g.user_id is set, else 401
- @admin_required decorator: validates is_admin, else 403
- Bot endpoints: @bot_login_required checks X-Bot-Token header against _bot_sessions in-memory dict
Bot Session Token System (/goyaljai page):
- GET /goyaljai → Jinja renders bot_goyaljai.html with {{ bot_token }} (short-lived token from _bot_sessions)
- Token issued at render time: secrets.token_urlsafe(32) stored in _bot_sessions dict {token: expiry}
- Bot JS sends X-User-Id: BOT_UID and X-Bot-Token: <token> on /api/stt, /api/voice-prompt
- Token expires on server restart (in-memory) → page reload gets fresh token
- _bot_sessions periodically pruned to prevent unbounded growth
Tailscale Network Layer:
- Server runs inside Ubuntu chroot on Android (PocketServer architecture)
- Exposed via Tailscale MagicDNS: ai-vps-goyaljai.tail98a210.ts.net
- CORS configured for /api/* → origins: "*" to allow embedded widget calls
- Gunicorn behind reverse proxy (supervisord-managed)
Admin Panel (/admin):
- Protected by @admin_required
- GET /api/admin/users → list all users with session counts and last active
- GET /api/admin/config → model config: fallback list, thinking model, TTS model
- POST /api/admin/models → update model routing config
- GET /api/admin/contacts → export contact list
- Audit log: all admin actions written to structured log
Show token lifecycle, refresh flows, bot token issuance/expiry, CORS boundaries, and failure/401 paths.
Draw a detailed high-level architecture diagram for the multimodal input processing pipeline.
Components and flows:
File Upload & Storage:
- User drags/pastes file into chat input → JS reads as ArrayBuffer
- POST /api/files/upload (multipart/form-data) with file + session_id
- files.py validates MIME type whitelist: image/*, application/pdf, text/*, audio/*
- File stored on disk: uploads/{user_id}/{session_id}/{filename}
- File metadata stored in vector DB: {file_id, user_id, session_id, filename, mime_type, size, path, uploaded_at}
- Returns {file_id, url} to client
- GET /api/files/{file_id} serves file with user ownership check
Image Handling:
- Images inlined into Gemini messages as Part(inline_data=blob(base64, mime_type))
- Resized client-side to max 1024px before upload (Canvas API)
- generate() / stream_generate() accepts images[] param → appended to user message parts
- Model: gemini-2.0-flash (vision-capable) selected automatically
PDF Processing:
- PDF uploaded → stored at uploads path
- At inference time: pdf.py extracts text per-page via PyMuPDF
- Extracted text chunked (512 tokens/chunk, 64 token overlap)
- Each chunk embedded and stored in vector DB alongside original file_id
- Top-K chunks retrieved by semantic similarity to user query
- Retrieved chunks injected into system prompt as "Document context:"
Audio File Upload (non-voice):
- Uploaded audio → transcribed via /api/stt offline path (Gemini transcribe_audio)
- Transcript stored as message context, audio file retained for reference
File Management:
- GET /api/files?session_id=X → list files for session
- DELETE /api/files/{file_id} → removes disk file + vector DB metadata
- Admin: GET /api/admin/files/stats → total storage used per user
Context Assembly with Files:
- prompt_engine.py: get_file_context(session_id, query) → semantic search over file chunks
- Returns top-K chunks formatted as XML: <document name="file.pdf" page="3">...</document>
- Token budget shared between conversation history and file context
Show upload flows, chunking/embedding pipeline, retrieval at inference time, and storage layers.
Draw a detailed high-level architecture diagram for the AI image and video generation pipeline.
Components and flows:
Image Generation (Gemini Imagen):
- User types prompt in chat with /image prefix or uses image generation UI panel
- POST /api/image/generate {prompt, aspect_ratio, model}
- app.py calls gemini_generate_image(prompt, model="imagen-3.0-generate-001")
- gemini.py sends request to Gemini Imagen API via google-generativeai SDK
- Returns base64 PNG bytes
- Image saved to uploads/{user_id}/generated/{uuid}.png
- Metadata stored in vector DB: {image_id, user_id, prompt, model, created_at, path}
- Response sent as data URL to client, rendered inline in chat bubble
- Client can download or share generated image
Video Generation (Veo 2):
- User triggers video generation (prompt or image-to-video)
- POST /api/video/generate {prompt, image_url (optional), duration_seconds}
- gemini.py: generate_video() calls Veo 2 API using _GEMINI_API_KEYS rotation
- Text-to-video: Operation polled every 5s until complete (async job)
- Image-to-video: same poll loop with source image as Part
- Operation name stored in vector DB with status: {op_id, user_id, prompt, status: pending/complete/failed}
- Polling: GET /api/video/status/{op_id} → checks operation state
- On complete: video bytes downloaded, stored at uploads/{user_id}/videos/{uuid}.mp4
- Video served via GET /api/video/{op_id} with range request support (streaming playback)
- Client shows progress bar during generation (SSE progress events)
API Key Management:
- _GEMINI_API_KEYS: list of 3 Gemini API keys hardcoded in gemini.py
- Round-robin key rotation on each API call (key_index = call_count % len(keys))
- On 429 (rate limit): skip to next key immediately, log warning
- On 403 (quota exceeded): mark key as degraded for 1hr, skip in rotation
- ELEVENLABS_API_KEY, ELEVENLABS_API_KEY_2: env vars, tried in order for TTS
TTS as a Generation Task:
- /api/tts endpoint: ElevenLabs key1 → key2 → Gemini TTS fallback
- Gemini TTS: generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-preview-tts
- Returns PCM audio converted to WAV by gemini.py (wave module header written manually)
- Streamed back to client as audio/wav or audio/mpeg
Admin Generation Controls:
- GET /api/admin/models → returns current model routing config
- POST /api/admin/models → update which Imagen model, Veo version, TTS model used
- Admin can add/remove TTS voice models via /api/admin/models/tts_models endpoints
Show all API call paths, key rotation logic, async polling, storage flows, and client rendering.