Pipeline-as-Config: structural model dispatch (#2114)

[justinchuby]

Summary

Implements the Pipeline-as-Config redesign from #2114 (PR1–5), by generalizing the existing DecoderOnlyPipeline executor rather than introducing the greenfield classes the issue sketched — "refactor not rewrite." The proposed PipelineExecutor/MultiSessionPipeline already effectively exist as DecoderOnlyPipelineState/DecoderOnlyPipelineModel.

Addresses #2114.

What's included

PR	Change
PR1	Config v2 schema: Config::version + Config::Pipeline; SAX v2 parse; TranslateV1ToPipeline / LowerPipelineToModel; pipeline_presets.*
PR2	Structural CreatePipeline() delegated from CreateModel when a pipeline is present; legacy model.type dispatch preserved as the ClassifyLegacyRoute oracle; v2 tokens/generation/metadata lowering
PR3	PipelineFlow (init/step/final phases, explicit dataflow[], DFS cycle detection, 10-stage guard) + Finalize hook
PR4	Plugin escape hatch: opaque C-ABI plugin_api.h + plugin_loader, gated behind USE_GENAI_PLUGINS (OFF by default)
PR5	ClassifyStructuralRoute replaces model.type dispatch for CP2/CP3/CP4, guarded by a zero-regression gate asserting structural == legacy for every in-tree fixture

Backward compatibility

• model_type.h predicates are retained — every one still has a live non-dispatch caller (v1→v2 translator, generators, kv_cache, RNNT context-length guard, legacy oracle). Only the dispatch role was removed. This is shrink-in-role, not the issue's "delete the file."
• All checked-in genai_config.json fixtures route identically (PipelineDispatchTests gate).
• The new qwen2-5-vl-pipeline gate fixture surfaced and fixed a real divergence: has_vision now recognizes the vision.pipeline[] shape (not just vision.filename).

Testing

• unit_tests: 76 passed / 0 failed / 21 skipped (skips are env-gated: TensorRT-RTX, StreamingASR, Parakeet).
• New suites green: PipelineConfigTests, PipelineFlowTests, PluginLoaderTests, PipelineDispatchTests.
• gpt2-fp32 + lfm2-fp32 e2e (CPU + CUDA) pass.
• Plugin ON-path (USE_GENAI_PLUGINS=1) compiles.

Not yet exercised end-to-end (reviewer attention welcome)

• Multi-stage VLM RunStage init/step loop and Finalize final-stage path — no in-tree generic decoder.pipeline[] VLM fixture with ONNX weights; behavior-preservation rests on the dispatch-equivalence gate + reasoning, not a running multi-session VLM.
• Downloaded-model Python tests and TensorRT/StreamingASR paths (environment-gated).

Descoped to v2.1+ (per maintainer comments in #2114)

TTS single_pass, diffusion denoising, RNNT loop strategies as plugin, when: "final" vocoder, repeat/counter. Encoder-decoder remains Whisper/Marian (routed structurally, not unified into the executor).

🤖 Draft — opening for early architectural review given the e2e caveats above.

[titaiwangms]

Review from a model-builder / CP6 perspective

I went through this with a focus on the producer↔consumer contract (this PR is the C++ config consumer; the Python model builder is the producer that would emit these configs). I traced schema → parser → router → executor against the diff. Net takeaway: today the v2 pipeline block is effectively a routing/metadata overlay — the real execution source of truth is still the legacy model.* structures. That's consistent with the honest config.h note ("default consumers continue to read config.model.*"), but the RFC/examples read as if pure-v2 is runnable, which I don't think it is yet. Details below, grounded in the diff.

Critical

1. v2 pipeline is not lowered into executable stages. LowerPipelineToModel only backfills session filenames + derives context_length; it doesn't materialize flow[]/dataflow[] into model.decoder.pipeline[]/output_names_forwarder, nor lower state.*. PipelineFlow sizes stage_phase_ to model.decoder.pipeline.size() and merely refines the phase of pre-existing legacy stages. So a pure-v2 multi-stage config (e.g. examples 03-vlm-per-image, 06-multimodal-single-pass) routes correctly but has zero stages to execute. A producer would still have to emit the full legacy decoder.pipeline[] + per-session inputs/outputs.

2. KV-cache / position inputs still read model.decoder.inputs.*, not pipeline.state. kv_cache.cpp / position_inputs.cpp were not switched to read pipeline.state.kv_cache / position_ids, and LowerPipelineToModel doesn't lower those names. A pure-v2 config that omits model.decoder.inputs would hit empty KV/position names. Relatedly, Whisper cross-attention names (cross_past_key_names, …) have no home in Config::Pipeline::State::CrossCache, so encoder-decoder cross-cache can't be expressed in v2.

3. pipeline.strategy / roles appear orphaned from the load path. ClassifyStructuralRoute and the Generator constructor never read pipeline.strategy; SpeculativeDecoder is constructed only in speculative_decoding_tests.cpp and exposed via no public API. So a config carrying a valid strategy block, loaded through the normal OgaModel/Generator path, would silently fall back to ordinary decoding (no draft, no speedup, no error). roles / verify.session / draft.session are parsed but never consumed or validated. (Design §3.3 describes a LowerPipelineToModel strategy branch that isn't implemented — that seems to be the root cause.)

Major

4. init/step gating still uses run_on_prompt / run_on_token_gen, not flow.when. flow.when only drives the final phase; for init/step the executor reads the legacy flags. A producer must keep both in sync.
5. dataflow drops the source session. explicit_wires_ is keyed by (to_session, to_tensor) and stores only from_tensor, then resolves against ortvalue_store_ by bare tensor name. The schema endpoint format is session.tensor, but semantics collapse to tensor — so intermediate tensor names must be globally unique across sessions, which isn't documented.
6. flow/dataflow endpoints aren't validated. Dangling or misspelled from/to/run references are silently ignored rather than rejected at load time.
7. Structural routing fails open. ClassifyStructuralRoute returns DecoderOnly as a catch-all for residual configs, whereas legacy dispatch returned Unsupported for unknown model.type — an unknown/misspelled decoder config now loads silently as a plain decoder.
8. variable_resolution and pure-v2 session_options are parsed but not consumed. CreateVisionState needs flow[].variable_resolution to pick Pixtral vs Qwen, but a pure-v2 Pixtral can't express it; v2 pipeline.sessions[].session_options (providers/provider_options) aren't lowered, so they're effectively ignored.
9. The equivalence gate is fixture-coverage, not a soundness proof. PipelineDispatchTests asserts structural == legacy over a hardcoded in-tree fixture list; it can't cover structurally-identical-but-different models outside that set (the central "detect, don't declare" bet).

Docs / clarity (would mislead a producer)

• The grammar logit-op example documents grammar/stateful fields that the parser discards (no struct fields) — copying it yields silently unconstrained output.
• 07-prefill-decode is a v1 config presented as the prefill/decode how-to; there's no v2 equivalent for that topology.
• No required-vs-optional field reference table (e.g. tokens.eos, generation.max_length are effectively required but undocumented as such).
• pipeline.plugin (model construction) vs pipeline.controller (decode loop) are easy to confuse and only plugin appears in an example.
• sessions serializes as a JSON object keyed by name, but config.h declares std::vector<Session> with no hint — a producer building from the header would emit an array and fail to parse.

Nice work

• ClassifyLegacyRoute vs ClassifyStructuralRoute + the equivalence test is a solid regression-safety pattern.
• PipelineFlow's DFS cycle detection + 10-stage guard are good load-time guardrails.
• The greedy speculative loop in speculative_decoder.cpp looks token-for-token correct under greedy (accept-longest-prefix + always-commit-≥1 + rollback re-seat), and unsupported producers/acceptance/tree all throw rather than silently misbehaving.

Questions

• Is the Python builder expected to emit native v2, or keep emitting v1 and rely on TranslateV1ToPipeline as the canonical path?
• Are flow/dataflow/state meant to be lowered into executable stages (and KV/position consumers switched to read pipeline.state), or is v2 a routing/metadata overlay for now?
• Is speculative-via-config in scope for this PR, or explicitly prototype-only?
• For the untested init/step/final + dataflow[] path: would you prefer a synthetic N-session toy-ONNX fixture or a VLM-shaped one, and does structural detection accept initializer-only dummy graphs? Happy to contribute that fixture on a branch stacked on this PR.