Course idea — 402 Route the loop: price/performance inference routing for agents

[fenar]

Premise: "routing" is two different problems, and the OSS ecosystem has already split cleanly along the blueprint's three tiers. This course is an integration and measurement course, not a build — every component exists; we teach composition and produce numbers nobody has published.

The stack (all CNCF-track OSS, composing by design):

• vLLM Semantic Router (Tier 1+2): Envoy ExtProc “Mixture-of-Models” layer — BERT intent/domain classification picks the cheapest capable model, decides reasoning on/off per request (When to Reason), and carries PII detection + jailbreak filtering + semantic caching.
• llm-d (Tier 3): CNCF Sandbox scheduler within each model's replica pool — prefix/KV-cache-aware balancing (kv-cache wins), predicted-latency scheduling (GA in v0.7), P/D disaggregation. It deliberately never chooses between models — which is exactly why the two projects stack: semantic-router ships official installs for Envoy AI Gateway / Gateway API Inference Extension, the same machinery llm-d builds on.
• RouteLLM (method): preference-trained win-rate thresholds, ~85% cost cut at 95% quality on chat benchmarks — our evaluation playbook, not a production component.
• Arch/Plano (counterpoint): routing-LLM-with-policies instead of classifiers; one comparison lab.

Why agent traffic is the original contribution — all published routing numbers are for chat:

1. Intra-episode heterogeneity: one 301 episode holds many cheap mechanical steps (tool formatting, JSON extraction) around a few hard ones (diagnosis, plan synthesis). Chat routes per conversation; agents pay off per step.
2. Amplified prefix-cache affinity: agent loops re-send growing context every step — llm-d's prefix-aware scheduling should win more on agent traffic than chat. Measurable with our tapes (KV hit rate, TTFT).
3. The privacy tier is the telco story: PII-classified steps pinned on-cluster (subscriber data in 202, topology in 301) — hybrid-cloud routing with wire-capture evidence.
4. Synergy with 301/302: Preference-tune the remediation-risk scorer from human approval-gate decisions #19: RouteLLM-style calibration needs preference data; our approval-gate logging produces it from real operator decisions.

Labs:

1. Baseline: replay 101/201/301 tapes through a single frontier model; record cost-per-resolved-episode.
2. Tier 2: Envoy AI Gateway + Semantic Router on RHOAI; per-step small-vs-frontier routing; target ≥50% cost cut at ≥95% task success.
3. Tier 3: llm-d under the on-cluster pool; prefix-cache hit rate + TTFT vs naive scheduling on loop traffic.
4. Tier 1: PII steps never leave the cluster; wire-capture evidence per QA conventions.
5. Comparison: classifier (semantic-router) vs routing-LLM (Arch) vs static rules (LiteLLM) on identical tapes.

Non-goal: building a router. The blueprint claims routed inference is cheaper without being worse — this course proves or disproves it on agent workloads, with evidence.