Skip to content

0006-voice-and-vision-local-first.md

Latest commit

 

History

History
15 lines (10 loc) · 3.4 KB

0006-voice-and-vision-local-first.md

File metadata and controls

15 lines (10 loc) · 3.4 KB
Raw

ADR 0006: Voice + vision, reclassified from speculative to built - local-first, free-tier by construction

Status: Accepted

Context: docs/ROADMAP.md originally filed voice and vision under "v2 (speculative) - research spikes before commitment," alongside genuinely harder things like voice cloning and live multi-speaker diarization. That triage was correct for those specific capabilities, but conflated them with plain speech-to-text, text-to-speech, and image OCR/description - all of which are mature, free, offline-capable technology today, not research. Zaid explicitly asked to reclassify this and build it now, so this ADR captures the "today's tech, not speculative" line, per the project's own critical-thinking mandate.

Decision:

  • Speech-to-text: faster-whisper (CTranslate2-backed Whisper), CPU, int8 compute, lazy-loaded on first use exactly like LocalEmbedder in router/providers.py - free, offline, no API key, model auto-downloads from Hugging Face on first run (same pattern as sentence-transformers).
  • Text-to-speech: pyttsx3, which drives the OS's native voices (SAPI5 on Windows) - zero download, zero API key, output saved to a .wav file rather than played directly, so it's a testable artifact instead of a side effect.
  • Vision - OCR ingestion: pytesseract + Pillow extend memory/ingest.py's read_file() so image files (.png/.jpg/.jpeg/.bmp/.tiff/.gif) become ingestable exactly like .pdf already is. Tesseract's OCR binary is a separate, external, non-pip-installable dependency - handled exactly like Ollama: optional, not auto-installed, fails with one clear actionable message (VisionError, pointing at the installer) instead of a stack trace, when missing.
  • Vision - Q&A: GeminiProvider.describe_image() sends image bytes + a prompt to Gemini's multimodal endpoint; ModelRouter.describe_image() delegates to it. This path is Gemini-only - Ollama has no equivalent here - and raises the same style of clear RouterError ("no GEMINI_API_KEY") as complete() does when unconfigured, rather than a special case.

Explicitly not built, and still correctly filed as speculative: voice cloning, real-time multi-speaker diarization, and any vision capability beyond OCR-ingest + single-image Q&A (e.g. live video, face recognition). Those remain genuinely harder and lower-priority; this ADR does not reclassify them.

Consequences: New dependencies (faster-whisper, pyttsx3, pytesseract, Pillow, python-multipart) add real install weight (ctranslate2 alone is ~19MB) but all install cleanly via pip on Windows; none require anything beyond what's already free-tier. Tests never touch a real model, real audio, or a real Tesseract binary - faster_whisper.WhisperModel and pyttsx3.init/pytesseract.image_to_string are mocked at their import boundary, keeping the suite fast and CI-safe (mirrors how Gemini/Ollama are always mocked in tests). The TTS->STT round trip was verified live with real local models (no mocks, no external audio/API key needed) as a self-contained proof: text synthesized to a real .wav, transcribed back, and the words matched exactly. OCR and Gemini-vision were verified live too, but both correctly degraded (Tesseract isn't installed on this machine; no Gemini key is configured yet) - proving the failure paths work, not the happy paths, which is the honest state of things until those two external dependencies are actually provided.