Proposal: conversation routing without a verifier

[panpan0000]

Design Proposal: Conversation Routing Without a Verifier

Status: draft.

Problem

The capability classifier's rule cards and the escalation judge are written for
agentic coding: every rule keys off a machine-verifiable outcome (validator,
harness, test, checker). Conversation tasks — chat, knowledge Q&A, writing,
summarization, tutoring, roleplay — have no verifier; the user is the only
judge. Such a request lands on primary_rule=none, capability_boundary=unmatched,
so the classifier emits a p_solve with no grounding, and the escalation judge
has no tests or builds to read. Routing is effectively a coin flip.

Why not just add a conversation rule card

The capability card is really a "verifiable-outcome card": its SUCCESS
definition ("whole task correct, as judged by the final verifier") and its
p_solve calibration both assume a verifier. A conversation card needs a
different SUCCESS definition ("rated acceptable by a reference evaluator") and
its own calibration set. Folding that into the same p_solve scale would
corrupt the coding calibration.

Proposal

Three layers, in order of cost:

1. Value prop first. Decide what efficient/capable means for conversation
   (draft vs. polished, or correct vs. incorrect). Everything else depends on it.
2. Cheap default + regret signal. Do not forecast difficulty from the first
   message. Route conversation by a deployment-level default, and escalate on
   in-session user feedback (corrections, "wrong", "rephrase", "not what I
   asked") — the user is a free, reliable verifier.
3. Only if a prior classifier is justified: a separate mode = conversation
   (mirroring capability/escalation/custom), with its own prompt+schema and
   rules grounded in instruction-visible failure modes (fact faithfulness,
   multi-step reasoning, exact format adherence, high-stakes domain, long-doc
   fidelity, multilingual). Calibrate against LLM-as-judge labels — no verifier,
   no calibration.

Prototype through the existing custom classifier mode before promoting to a
first-class mode.

Non-goals

• No changes to the coding capability card or escalation judge.
• No difficulty guessing; the request-fit length gate is domain-agnostic and
  stays separate.

Open questions

1. Does conversation routing want the efficient/capable binary at all, or a
   different target set (e.g. draft/careful)?
2. Which conversation classes first (chat, Q&A, writing, tutoring)?
3. LLM-as-judge or human labels for calibration?