Chat: RAG grounding layer + toggle

[mikkisguy]

Parent

Chat Interface — Design Spec (docs/superpowers/specs/2026-06-19-chat-interface-design.md)
App Security Baseline — Design Spec (docs/superpowers/specs/2026-06-19-app-security-baseline-design.md)

What to build

The RAG grounding layer: retrieve relevant items from the ShadowBrain SQLite
and inject them as context, toggleable per thread and per send. When off, the
chat is plain general chat (no knowledge-base context). The retrieval respects
the two-level visibility model: is_hidden items are AI-OK by default;
is_private items are excluded unless the current thread has opted in via a
per-thread "Include private in AI" control.

• lib/chat/retrieval.ts — given the latest user message, run FTS retrieval
  reusing the Phase 1 search helper / sanitizeFts5Query, with tag/type
  filters. Default behavior: include is_hidden = 1 items (they are AI-OK);
  exclude is_private = 1 items unless the thread has
  include_private_in_ai = 1. Return top-K items (CHAT_RAG_TOP_K, default
  8). Render a ## Retrieved context block (title, type, snippet, tags).
• Wire into /api/chat: when grounded is true, prepend the context block as
  a system message before forwarding to the provider.
• grounded comes from the thread default (chat_threads.grounded) and is
  overridable per send.
• The "Include private in AI" toggle: a new column
  chat_threads.include_private_in_ai INTEGER NOT NULL DEFAULT 0 (per-thread
  setting, off by default) plus a per-send override in the chat UI. When on,
  the RAG layer calls the read helpers with includePrivate = true for that
  thread / message.
• Config: CHAT_RAG_TOP_K.

Acceptance criteria

• retrieval.ts returns top-K items via FTS
• Default: is_hidden = 1 items are included in RAG context (AI-OK)
• Default: is_private = 1 items are excluded from RAG context
• When the thread's include_private_in_ai = 1 (per-thread or per-send override), is_private items are included
• grounded=true injects a ## Retrieved context system block; grounded=false is plain chat
• The include_private_in_ai toggle is persisted on the thread and overridable per send
• Empty retrieval → proceeds without context (never blocks or errors)
• Works for both opencode-go and hermes targets (context injection is provider-agnostic)
• Tests: retrieval (FTS + is_hidden include + is_private exclude-by-default + is_private include on opt-in), grounded vs ungrounded injection, per-thread toggle

Blocked by

• Chat: provider abstraction + streaming hub + persistence + minimal UI #48 — Chat: provider abstraction + streaming hub + persistence + minimal UI
• Full-text search API #11 — Full-text search API (FTS search helper)
• Database layer with schema and migrations #2 — Database layer (FTS5 virtual table + triggers)
• Security: two-level visibility (is_hidden + is_private) #54 — Security: two-level visibility (for the includeHidden / includePrivate read-helper options)

References

• docs/superpowers/specs/2026-06-19-chat-interface-design.md — §Components (retrieval.ts), §Architecture, §Configuration, §Data Model, §Security
• docs/superpowers/specs/2026-06-19-app-security-baseline-design.md — §2 (two-level visibility)