InferenceRequest.async_exec(decoupled=True) intermittently raises `RuntimeError: Invalid argument` under concurrent in-flight BLS to a decoupled model

[protonicage]

Environment

• Triton Inference Server version: 2.70.0 (NVIDIA Release 26.06, build 55746352)
• Python backend: triton_python_backend_utils from libtriton_python.so (build 2025-06-24)
• Python runtime: 3.12.3
• Backends loaded: identity, python, pytorch, vllm
• GPU: NVIDIA A100-SXM4-80GB, driver 595.71.05 (CUDA 13.2)
• Topology hosting the bug:

  ensemble   (Python backend, KIND_CPU, 8 instances, async BLS orchestrator)
    ├─ ...
    ├─ ...
    └─ whisper_vllm (vllm backend, decoupled)  <-- BLS target
  

Summary

When an async Python model (async def execute) issues BLS requests to a decoupled model using
pb_utils.InferenceRequest.async_exec(decoupled=True), and more than a handful of these are in flight
concurrently, the call intermittently fails with:

RuntimeError: Invalid argument

raised from inside concurrent/futures/thread.py (the worker-thread self.fn(*self.args, **self.kwargs)
dispatched by async_exec). The failure is intermittent — it appears to depend on the number of
simultaneously in-flight decoupled BLS streams, not on the request contents or shapes.

Expected behavior

Per the Python backend docs (Business Logic Scripting section, "Decoupled Models" -> "Using BLS with
Decoupled Models"), the documented pattern is:

import triton_python_backend_utils as pb_utils
import asyncio

class TritonPythonModel:
    async def execute(self, requests):
        inference_request = pb_utils.InferenceRequest(
            model_name='whisper_vllm',
            requested_output_names=['text_output'],
            inputs=[...])

        infer_response_awaits = [
            inference_request.async_exec(decoupled=True)
            for _ in range(4)
        ]
        async_responses = await asyncio.gather(*infer_response_awaits)

        for infer_responses in async_responses:
            for infer_response in infer_responses:
                assert not infer_response.has_error()
                ...

This should return one Awaitable per call, each resolving to an iterator of the decoupled responses,
and should be safe to fan out concurrently.

Actual behavior

With the exact pattern above, the code intermittently crashes with RuntimeError: Invalid argument:

File ".../whisper_vllm.py", line 240, in _run_single_whisper
    iterator = await request.async_exec(decoupled=True)
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
RuntimeError: Invalid argument

Observations:

• RuntimeError surfaces from a ThreadPoolExecutor worker thread — i.e. async_exec internally
  schedules the blocking exec onto a worker thread and returns a Future, and the wrapped C++ call
  returns EINVAL.
• Raising concurrency increases the probability of failure; a single sequential call rarely fails.
• Workaround that is reliable in 2.70.0: run the synchronous request.exec(decoupled=True) inside
  asyncio.to_thread(...) with a bounded semaphore (i.e. offload the documented stable sync API to a
  worker thread instead of using async_exec). This confirms the problem is isolated to the
  async_exec(decoupled=True) path, not to concurrency against the decoupled model in general.

# reliable workaround in 2.70.0
iterator = await asyncio.to_thread(request.exec, decoupled=True)
for response in iterator:
    ...

Minimal reproducer

A minimal two-model repo pair ("orchestrator" = async Python backend model, "leaf" = decoupled Python
backend model) that fans out N concurrent async_exec(decoupled=True) BLS calls would reproduce the
intermittent RuntimeError. I have not yet packaged a fully standalone repo, but the essential
ingredients are:

1. A decoupled leaf Python model whose execute writes a response and returns None
   (model_transaction_policy: { decoupled: True }).
2. An async orchestrator Python model that, per request, issues several
   await asyncio.gather(*[req.async_exec(decoupled=True) for _ in range(N)]) with N >= 8.
3. A client that bursts many concurrent orchestrator requests.

If useful I can provide the full model definitions, but the three ingredients above drive 100% of the
behaviour.

Impact

• Any production policy that relies on async Python-model orchestration over a decoupled backend
  (e.g. an ensemble/stub that diarizes -> transcribes via a streaming/decoupled ASR or LLM) is at risk
  of nondeterministic per-request failures from this single API path.
• The failure is a hard RuntimeError that aborts the whole BLS request (all gathered sub-requests),
  not a graceful per-response error.

[adityaanikam]

Traced this into python_backend, since that's where async_exec and exec both actually live rather than here. Both calls funnel into the identical InferRequest::Exec() in infer_request.cc, exec() runs it on the calling thread, async_exec() dispatches the same call through loop.run_in_executor onto Python's default ThreadPoolExecutor. The IPC layer underneath, the boost::interprocess queue with its mutex and semaphores, looks correctly synchronized for concurrent producers from what I can read.

The one shared, decoupled specific resource I found that looks racy under enough concurrent load is response_iterator_map_ in pb_stub.cc, keyed by a raw void* taken from the response payload's address, with cleanup happening asynchronously on a separate callback. If a payload gets freed and its address reused before that cleanup runs, a new concurrent decoupled stream could collide with a stale map entry, classic ABA shape, and it would get worse exactly as concurrency increases, matching what you're seeing. I can't pin the actual EINVAL past that though, it's coming from somewhere in boost::interprocess or the underlying pthread calls, outside what's in this repo. One thing worth double checking on your end: your workaround uses a bounded semaphore where the failing path uses an unbounded gather, so the comparison isn't apples to apples, worth confirming the crash still reproduces with the async path bounded to the same concurrency your workaround uses before ruling that out as a factor.