Skip to content

Latest commit

 

History

History
9 lines (5 loc) · 2.14 KB

File metadata and controls

9 lines (5 loc) · 2.14 KB

ADR 0004: Multi-provider verification via local-embedding agreement, not a paid arbiter model

Status: Accepted

Context: The v0.3 roadmap calls for a "richer router" with confidence/verification and eventual paid-model support. A paid third provider can't be built or tested honestly right now - no paid API key is configured (.env policy: never invented, always user-supplied), and free-tier-by-default is a locked principle. The real, buildable slice is using the two providers already wired (ADR 0002: Ollama + Gemini) to cross-check each other when both happen to be available.

Decision: ModelRouter.complete_with_verification() calls every currently-available chat provider (not just the first healthy one, unlike complete()) and returns a VerifiedCompletion: a primary answer (same priority order as complete()), the alternate answer(s), and a disagreement flag. Agreement is measured by cosine similarity between the local embeddings (already-free, already-offline via sentence-transformers) of each answer's text against the primary's, using a fixed threshold (0.6). No new dependency, no network call beyond the providers already being queried. rag.pipeline.ask() takes a verify: bool = False parameter that switches between complete() and complete_with_verification(); both CLI (ask --verify) and API (AskRequest.verify) expose it as an explicit per-call opt-in, not a global mode - verification costs an extra provider call, so it shouldn't run by default.

Consequences: With only one provider configured (the common case today - Gemini or nothing), verification degrades to a plain single completion with disagreement=False and no alternates - it never fails harder than complete() does. Adding a paid provider later is exactly a new ChatProvider in the list (no change to this logic), and it would immediately start participating in verification. The 0.6 similarity threshold is a heuristic tuned by eyeballing agreeing vs. disagreeing text pairs, not a calibrated confidence score - documented here so it isn't mistaken for one; revisit if false-positive/negative disagreement flags turn out to be common in real use.