Bound LLM concurrency across concurrent runs in a host-owned VM

[andreasronge]

Problem

live_provider_tasks is enforced by one RunState, while ReqLLM's Finch instance and its destination pools are shared across the VM. Configuring one destination pool as count: 1, size: live_provider_tasks proves that a single isolated run cannot oversubscribe that pool, but it does not bound aggregate demand from concurrent runs.

Two runs may each admit eight provider tasks and present sixteen checkouts to a pool of eight connections. Direct embedding calls and other ReqLLM operations may also bypass a run's Dispatcher reservation.

Increasing Finch's default five-second pool_timeout only moves failure later. No finite static timeout guarantees service under unbounded concurrent-run demand, and any wait is also bounded by the provider operation and run deadlines.

This is the normal shape of hosting PtcRunner as a service, not a command-VM pool-geometry defect.

Required decision

Choose an explicit VM-wide contract for host-owned runtimes:

1. a Kernel/host-owned global LLM admission gate aligned with configured pool capacity; or
2. a host-declared aggregate concurrency and destination-pool capacity contract with bounded queueing.

A global admission gate is the stronger default: queue before entering Finch, bound the wait by the caller's remaining deadline, and make saturation observable through Kernel vocabulary instead of an accidental Finch checkout exception.

Command-owned versus host-owned

• Command-owned VM: Use one adequately sized Finch pool for command-owned LLM concurrency #1287 can configure ReqLLM before application startup for the command's installed concurrency ceiling.
• Host-owned VM: :req_llm is already running and its lifecycle/configuration belongs to the embedding host. Per-run effective limits must not silently determine VM-lifetime pool geometry.
• If the first manifest narrows live_provider_tasks, that narrower value must not permanently size the shared application for later runs.

Investigation

• Inventory every path that reaches ReqLLM.Finch outside Dispatcher reservations, including direct PtcRunner.LLM calls, connectivity operations, embeddings, and other ReqLLM consumers.
• Confirm which traffic shares the ReqLLM Finch instance and destination pool.
• Determine how the host can attest or expose effective pool capacity after :req_llm has started.
• If Finch pool_timeout remains relevant, inject it per request and clamp it to the remaining operation/run deadline rather than using an arbitrary generous constant.

Acceptance criteria

• A deterministic concurrent-run test demonstrates bounded behavior with aggregate demand above one run's limit.
• No per-run value silently reconfigures or claims to describe VM-global capacity.
• Saturation has a bounded, documented outcome.
• Command-owned and host-owned responsibilities are documented separately.
• Direct adapter paths outside Dispatcher reservation are either admitted by the same global mechanism or explicitly documented as host responsibility.

Related

• Use one adequately sized Finch pool for command-owned LLM concurrency #1287 — command-owned Finch pool geometry and the original single-run incident.
• llm-request: select the model per call, so one run can use several #1240 — multiple aliases may target different destination pools, but alias quotas and host-level connection capacity remain orthogonal.