fix(generation): flag high-confidence answers that cite nothing

[leyder-ben]

Summary

Found while poking at the Phase 4 web UI: real /api/query calls kept coming back with a self-reported confidence the response gave no way to trust or audit — high with zero sources, or a downgraded confidence with no explanation. Three commits, each closing one gap in that trust chain:

1. fix(generation): flag high-confidence answers that cite nothing
The model can draw on retrieved vault content in say_this without populating used_source_chunk_ids — the system prompt forbids this but doesn't reliably prevent it (confirmed non-deterministic: identical query/context, repeated calls, citations present on some runs and absent on others). Rather than trying to verify what the model used (lexical/relevance matching against say_this — more surface area, fuzzy thresholds, a second system built to catch the first one lying), this adds one cheap deterministic check: self-reported HIGH confidence with an empty used_source_chunk_ids gets downgraded one level and flagged in limitations. A self-reported MEDIUM/LOW with no citations is left alone — that's the model already hedging honestly (e.g. "I'm not aware of this in your notes"), and re-flagging it would misrepresent an honest answer as a suspected violation.

2. feat(api): log confidence and limitations on query_runs
query_runs captured retrieval metadata (chunk ids, scores, selected sources) but not the final confidence/limitations the response actually carried — made it impossible to audit after the fact whether a given historical query was downgraded, without re-running it. Adds both columns, populated from the same AnswerResult the response body is built from. Migration 0003.

3. fix(generation): fall back to a generic limitation when a below-HIGH confidence is unexplained
Investigating the above surfaced a second, distinct gap: the model can self-report MEDIUM/LOW confidence with an empty limitations list — confirmed non-deterministic against the same live query repeated many times (sometimes explained, sometimes not). The prompt only requires an explanation for one specific case (an unsupported personal claim), not generally whenever confidence drops below HIGH. Same philosophy as commit 1: don't try to fix model prompt adherence, just refuse to let a degraded confidence stand with no stated reason. Fires only when limitations is genuinely empty, so it never overrides an existing downgrade's own explanation — confirmed live that both checks can fire together (model's own explanation + the citation-recall flag, stacked).

Root behavior not addressed here, flagged as a follow-up: why the model sometimes omits citations or explanations for content/confidence it demonstrably has opinions about isn't fixed, just made visible and auditable. Two directions worth considering later: a retry-on-incomplete-structured-output loop, or making these fields schema-required so an incomplete response fails parsing and forces a retry.

Test plan

• generation/test_service.py: 6 new tests across the three commits (flagged-and-downgraded case, honest-hedge guard rail, fallback-limitation case x2, well-cited-HIGH regression guard, no-double-flagging regression guard)
• test_query_api.py: updated to assert confidence/limitations are persisted to query_runs
• Full backend suite: 185/185 passing
• ruff format --check / ruff check: clean
• Live verification against the real backend + Ollama (gpt-oss:20b), repeated runs across multiple real queries (terraform drift, Helm, Kubernetes vs ECS, Ansible, Meridian, CI/CD) - confirmed via query_runs, post-fix, zero rows show a below-high confidence with empty limitations, and the previously-silent high/0-sources pattern no longer appears unflagged

🤖 Generated with Claude Code