Question about DALI + Triton: Dynamic Batching and Error Handling

[lokesh-kurre]

I'm using NVIDIA DALI together with Triton Inference Server (r23.07). My DALI pipeline includes an image decoding step and is deployed with Triton's dynamic batching enabled. For Testing purpose I have set max batching delay of 10 seconds.

If I send two concurrent requests to Triton—one with a valid image (supported format) and one with an invalid image (corrupted or unsupported format)—Triton's dynamic batching groups these requests together. However, I observe that only the request with the invalid image fails, while the request with the valid image completes successfully.

My expectation was that if any item in the batch fails decoding, the whole batch would fail and all requests in the batch would return errors. But this does not seem to be the case. Is this the intended behavior for DALI and Triton? Does error reporting always occur per input/request, even when dynamic batching is enabled and a single pipeline instance processes both valid and invalid samples?

For Failure Case I got below error (in triton side):

E0311 08:11:48.872766 450776 dali_model_instance.cc:46] Critical error in pipeline:
Error when executing Mixed operator decoders__Image encountered:
Error in thread 1: [/opt/dali/dali/operators/decoder/nvjpeg/nvjpeg_decoder_decoupled_api.h:616] [/opt/dali/dali/image/image_factory.cc:100] Unrecognized image format. Supported formats are: JPEG, PNG, BMP, TIFF, PNM, JPEG2000 and WebP.
Stacktrace (7 entries):
[frame 0]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0xc32fb) [0x7f414f7042fb]
[frame 1]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x8a604) [0x7f414f6cb604]
[frame 2]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7a26d) [0x7f4125e0426d]
[frame 3]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(dali::ThreadPool::ThreadMain(int, int, bool, std::string const&)+0x1e6) [0x7f414f7ea6c6]
[frame 4]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x6baa80) [0x7f414fcfba80]
[frame 5]: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f42f620eb43]
[frame 6]: /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f42f629fbb4]
. File:
Stacktrace (6 entries):
[frame 0]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0x67183e) [0x7f41259fb83e]
[frame 1]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7ac5a) [0x7f4125e04c5a]
[frame 2]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(dali::ThreadPool::ThreadMain(int, int, bool, std::string const&)+0x1e6) [0x7f414f7ea6c6]
[frame 3]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x6baa80) [0x7f414fcfba80]
[frame 4]: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f42f620eb43]
[frame 5]: /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f42f629fbb4]

Current pipeline object is no longer valid.

Thanks for your help!

[banasraf]

Hello @lokesh-kurre
Thanks for reporting the issue! The behaviour you're describing should not be the case. When requests are batched together, all of them should return an error if it occurs. Could you maybe run the Triton server with --log-verbose=2 flag and post here a full output of the server? It would be easier to inspect what is happening with those requests that way.

[lokesh-kurre]

Added sufficient max_queue_delay_microseconds (~5sec) so multiple requests can be clubbed together.

backend: "dali"
max_batch_size: 12

input [
    {
        name: "INPUT_BUFFERS"
        data_type: TYPE_UINT8
        dims: [ -1 ]
    },
    {
        name: "BUFFER_SIZES"
        data_type: TYPE_UINT32
        dims: [ 1 ]
    }
]

output [
    {
        name: "PADDED_PROCESSED_IMAGES"
        data_type: TYPE_FP32
        dims: [ -1, -1, -1 ]
    }
]

instance_group [
    {
        kind: KIND_GPU
        gpus: [0]
        count: 1
    }
]

dynamic_batching{
    max_queue_delay_microseconds: 5000000
    preserve_ordering: true
}





dali code:
import nvidia.dali as dali
from nvidia.dali import types
from nvidia.dali.plugin.triton import autoserialize

import config


@autoserialize
@dali.pipeline_def(
    batch_size=12, num_threads=6, device_id=0, exec_pipelined=False, exec_async=False
)
def pipe():
    image_buffers = dali.fn.external_source(name="INPUT_BUFFERS")
    buffer_size = dali.fn.external_source(name="BUFFER_SIZES")
    buffer_size = dali.fn.cast(buffer_size, dtype=types.INT32)
    image_buffers = dali.fn.slice(image_buffers, axes=0, end=buffer_size)
    raw_image_rgb = dali.fn.decoders.image(
        image_buffers,
        output_type=types.DALIImageType.RGB,
        device_memory_padding=25165824,
        device_memory_padding_jpeg2k=25165824,
        host_memory_padding=25165824,
        host_memory_padding_jpeg2k=25165824,
        device="mixed",
    )

    # # START: MTCNN PREPROCESS - BGR Image
    # padded_processed_images = dali.fn.color_space_conversion(
    #     raw_image_gray, image_type=types.DALIImageType.RGB, output_type=types.DALIImageType.BGR
    # )
    # padded_processed_images = dali.fn.cast(
    #     padded_processed_images, dtype=types.FLOAT
    # )
    # # END: MTCNN PREPROCESS - BGR Image

    # START: YOLO PREPROCESS (LetterBox Code) - RGB Image
    padded_processed_images = dali.fn.resize(
        raw_image_rgb,
        antialias=False,
        interp_type=types.DALIInterpType.INTERP_LINEAR,
        mode="not_larger",
        resize_y=config.YOLO_NORMALIZED_SHAPE[0],
        resize_x=config.YOLO_NORMALIZED_SHAPE[1],
    )
    padded_processed_images = dali.fn.crop(
        padded_processed_images,
        crop_h=config.YOLO_NORMALIZED_SHAPE[0],
        crop_w=config.YOLO_NORMALIZED_SHAPE[1],
        fill_values=[114, 114, 114],
        out_of_bounds_policy="pad",
    )
    padded_processed_images = dali.fn.transpose(
        padded_processed_images, output_layout="CHW", perm=[2, 0, 1]
    )
    # padded_processed_images = dali.fn.stack(padded_processed_images, padded_processed_images, padded_processed_images, axis=0, axis_name= "C", device="gpu")
    padded_processed_images = padded_processed_images / 255
    # END: YOLO PREPROCESS (LetterBox Code) - RGB Image

    return padded_processed_images

code ran:

def func(key):
    bio_exception_value = parse_bdr_resp(data[key]) #, biometric_types= 'face iris')
    payload_data = { "type": 'raw',  **bio_exception_value, "handle_bio_exception": True, "missing_biometrics": []}
    with session_context() as _sess:
        resp = _sess.post('http://10.10.240.21:12700/v2/models/face_embedding/infer', json={
            "id" : key,
            "inputs" : [ { "name" : "REQ_DATA", "shape" : [ 1 ], "datatype"  : "BYTES", "data" : [ json.dumps(payload_data) ] } ],
        })

with ThreadPoolExecutor(2) as executor:
    # _ = executor.map(func, ['only_iris_and_face'])                                        # case1
    # _ = executor.map(func, ['decoding_failed'])                                           # case2
    _ = executor.map(func, ['only_iris_and_face', 'decoding_failed'])              # case3

When only case1 is executed it successfully ran.
When only case2 ran it fails with prev reported error.

In case of case3 -> one sucess & one failure.
log:

I0320 08:18:34.314219 810908 http_server.cc:3452] HTTP request: 2 /v2/models/face_embedding/infer
I0320 08:18:34.314261 810908 model_lifecycle.cc:328] GetModel() 'face_embedding' version -1
I0320 08:18:34.314278 810908 model_lifecycle.cc:328] GetModel() 'face_embedding' version -1
I0320 08:18:34.321721 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f37f8318740] request id: only_iris_and_face, model: face_embedding, requested version: -1, actual version: 1
, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f37f8215598] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
override inputs:
inputs:
[0x0x7f37f8215598] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
requested outputs:
RESP_DATA

I0320 08:18:34.321757 810908 model_lifecycle.cc:328] GetModel() 'face_00_validation' version 1
I0320 08:18:34.321768 810908 model_lifecycle.cc:328] GetModel() 'face_01_landmark_detection_preprocess' version 1
I0320 08:18:34.321777 810908 model_lifecycle.cc:328] GetModel() 'face_02_landmark_detection_model' version 1
I0320 08:18:34.321769 810908 http_server.cc:3452] HTTP request: 2 /v2/models/face_embedding/inferI0320 08:18:34.321786 810908 model_lifecycle.cc:328] GetModel() 'face_03_preprocess' version 1
                                                                                                                                                                                                                I0320 08:18:34.321829 810908 model_lifecycle.cc:328] GetModel() 'face_04_model_preprocess' version 1
I0320 08:18:34.321873 810908 model_lifecycle.cc:328] GetModel() 'face_05_embedding_model' version 1
I0320 08:18:34.321893 810908 model_lifecycle.cc:328] GetModel() 'face_06_embedding_postprocess' version 1
I0320 08:18:34.321917 810908 model_lifecycle.cc:328] GetModel() 'face_embedding' version -1
I0320 08:18:34.321929 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f37f831bd40] request id: only_iris_and_face, model: face_00_validation, requested version: 1, actual version
: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f37f8026318] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
override inputs:
inputs:
[0x0x7f37f8026318] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
FACE_BUFFER_SIZES
FACE_INPUT_BUFFERS
FACE_TYPES
RESP_JSON
requested outputs:
FACE_BUFFER_SIZES
FACE_INPUT_BUFFERS
FACE_TYPES
RESP_JSON

I0320 08:18:34.321967 810908 model_lifecycle.cc:328] GetModel() 'face_embedding' version -1
I0320 08:18:34.322100 810908 python_be.cc:1263] model face_00_validation, instance face_00_validation_0_1, executing 1 requests
I0320 08:18:34.328281 810908 infer_request.cc:751] [request id: decoding_failed] prepared: [0x0x7f37f4003930] request id: decoding_failed, model: face_embedding, requested version: -1, actual version: 1, flag
s: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f37f4004068] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
override inputs:                                                                                                                                                                                      [218/1858]
inputs:
[0x0x7f37f4004068] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
requested outputs:
RESP_DATA

I0320 08:18:34.328350 810908 model_lifecycle.cc:328] GetModel() 'face_00_validation' version 1
I0320 08:18:34.328381 810908 model_lifecycle.cc:328] GetModel() 'face_01_landmark_detection_preprocess' version 1
I0320 08:18:34.328408 810908 model_lifecycle.cc:328] GetModel() 'face_02_landmark_detection_model' version 1
I0320 08:18:34.328437 810908 model_lifecycle.cc:328] GetModel() 'face_03_preprocess' version 1
I0320 08:18:34.328468 810908 model_lifecycle.cc:328] GetModel() 'face_04_model_preprocess' version 1
I0320 08:18:34.328495 810908 model_lifecycle.cc:328] GetModel() 'face_05_embedding_model' version 1
I0320 08:18:34.328522 810908 model_lifecycle.cc:328] GetModel() 'face_06_embedding_postprocess' version 1
I0320 08:18:34.328598 810908 infer_request.cc:751] [request id: decoding_failed] prepared: [0x0x7f37f4086450] request id: decoding_failed, model: face_00_validation, requested version: 1, actual version: 1, f
lags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f37f40869a8] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
override inputs:
inputs:
[0x0x7f37f40869a8] input: REQ_DATA, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
FACE_BUFFER_SIZES
FACE_INPUT_BUFFERS
FACE_TYPES
RESP_JSON
requested outputs:
FACE_BUFFER_SIZES
FACE_INPUT_BUFFERS
FACE_TYPES
RESP_JSON

I0320 08:18:34.328794 810908 python_be.cc:1263] model face_00_validation, instance face_00_validation_0_2, executing 1 requests
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "validation done"}
I0320 08:18:34.342379 810908 infer_response.cc:167] add response output: output: FACE_INPUT_BUFFERS, type: UINT8, shape: [1,555805]
I0320 08:18:34.342447 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 555805, addr 0x7f3ee8000090
I0320 08:18:34.342480 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_INPUT_BUFFERS, size 555805, addr 0x7f3ee8000090, memory type 1, type id 0
I0320 08:18:34.342650 810908 infer_response.cc:167] add response output: output: FACE_TYPES, type: BYTES, shape: [1]
I0320 08:18:34.342681 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 10, addr 0x7f3ee8087bc0
I0320 08:18:34.342706 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_TYPES, size 10, addr 0x7f3ee8087bc0, memory type 1, type id 0
I0320 08:18:34.342735 810908 infer_response.cc:167] add response output: output: FACE_BUFFER_SIZES, type: UINT32, shape: [1]
I0320 08:18:34.342765 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 4, addr 0x7f3ee8087bf0
I0320 08:18:34.342788 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_BUFFER_SIZES, size 4, addr 0x7f3ee8087bf0, memory type 1, type id 0
I0320 08:18:34.342816 810908 infer_response.cc:167] add response output: output: RESP_JSON, type: BYTES, shape: [1]
I0320 08:18:34.342843 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 344, addr 0x7f3ee8087c20
I0320 08:18:34.342868 810908 ensemble_scheduler.cc:555] Internal response allocation: RESP_JSON, size 344, addr 0x7f3ee8087c20, memory type 1, type id 0
I0320 08:18:34.342916 810908 ensemble_scheduler.cc:570] Internal response release: size 555805, addr 0x7f3ee8000090
I0320 08:18:34.342943 810908 ensemble_scheduler.cc:570] Internal response release: size 10, addr 0x7f3ee8087bc0
I0320 08:18:34.342968 810908 ensemble_scheduler.cc:570] Internal response release: size 4, addr 0x7f3ee8087bf0
I0320 08:18:34.342993 810908 ensemble_scheduler.cc:570] Internal response release: size 344, addr 0x7f3ee8087c20
I0320 08:18:34.343055 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f37f826e6f0] request id: only_iris_and_face, model: face_01_landmark_detection_preprocess, requested version
: 1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3a7000f008] input: INPUT_BUFFERS, type: UINT8, original shape: [1,555805], batch + shape: [1,555805], shape: [555805]
[0x0x7f37f826ec48] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
override inputs:
inputs:
[0x0x7f37f826ec48] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
[0x0x7f3a7000f008] input: INPUT_BUFFERS, type: UINT8, original shape: [1,555805], batch + shape: [1,555805], shape: [555805]
original requested outputs:
PADDED_PROCESSED_IMAGES
requested outputs:
PADDED_PROCESSED_IMAGES

I0320 08:18:34.343228 810908 python_be.cc:2230] TRITONBACKEND_ModelInstanceExecute: model instance name face_00_validation_0_1 released 1 requests
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "31036dc3-133a-419d-92d0-d6a64a70d2b2", "message": "validation done"}
I0320 08:18:34.344001 810908 infer_response.cc:167] add response output: output: FACE_INPUT_BUFFERS, type: UINT8, shape: [1,251977]
I0320 08:18:34.344078 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 251977, addr 0x7f3ee8087d80
I0320 08:18:34.344111 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_INPUT_BUFFERS, size 251977, addr 0x7f3ee8087d80, memory type 1, type id 0
I0320 08:18:34.344234 810908 infer_response.cc:167] add response output: output: FACE_TYPES, type: BYTES, shape: [1]
I0320 08:18:34.344266 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 14, addr 0x7f3ee80c55e0
I0320 08:18:34.344293 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_TYPES, size 14, addr 0x7f3ee80c55e0, memory type 1, type id 0
I0320 08:18:34.344323 810908 infer_response.cc:167] add response output: output: FACE_BUFFER_SIZES, type: UINT32, shape: [1]
I0320 08:18:34.344354 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 4, addr 0x7f3ee80c5610
I0320 08:18:34.344380 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_BUFFER_SIZES, size 4, addr 0x7f3ee80c5610, memory type 1, type id 0
I0320 08:18:34.344408 810908 infer_response.cc:167] add response output: output: RESP_JSON, type: BYTES, shape: [1]
I0320 08:18:34.344439 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 371, addr 0x7f3ee80c5640
I0320 08:18:34.344466 810908 ensemble_scheduler.cc:555] Internal response allocation: RESP_JSON, size 371, addr 0x7f3ee80c5640, memory type 1, type id 0
I0320 08:18:34.344516 810908 ensemble_scheduler.cc:570] Internal response release: size 251977, addr 0x7f3ee8087d80
I0320 08:18:34.344543 810908 ensemble_scheduler.cc:570] Internal response release: size 14, addr 0x7f3ee80c55e0
I0320 08:18:34.344568 810908 ensemble_scheduler.cc:570] Internal response release: size 4, addr 0x7f3ee80c5610
I0320 08:18:34.344594 810908 ensemble_scheduler.cc:570] Internal response release: size 371, addr 0x7f3ee80c5640
I0320 08:18:34.344647 810908 infer_request.cc:751] [request id: decoding_failed] prepared: [0x0x7f37f8291f80] request id: decoding_failed, model: face_01_landmark_detection_preprocess, requested version: 1, a
ctual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3888008b18] input: INPUT_BUFFERS, type: UINT8, original shape: [1,251977], batch + shape: [1,251977], shape: [251977]
[0x0x7f37f8270568] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
override inputs:
inputs:
[0x0x7f37f8270568] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
[0x0x7f3888008b18] input: INPUT_BUFFERS, type: UINT8, original shape: [1,251977], batch + shape: [1,251977], shape: [251977]
original requested outputs:
PADDED_PROCESSED_IMAGES
requested outputs:
PADDED_PROCESSED_IMAGES

I0320 08:18:34.344854 810908 python_be.cc:2230] TRITONBACKEND_ModelInstanceExecute: model instance name face_00_validation_0_2 released 1 requests
I0320 08:18:34.359229 810908 infer_response.cc:167] add response output: output: PADDED_PROCESSED_IMAGES, type: FP32, shape: [1,3,640,480]
I0320 08:18:34.359258 810908 ensemble_scheduler.cc:555] Internal response allocation: PADDED_PROCESSED_IMAGES, size 3686400, addr 0x7f3ea8000000, memory type 2, type id 0
I0320 08:18:34.359324 810908 ensemble_scheduler.cc:570] Internal response release: size 3686400, addr 0x7f3ea8000000
I0320 08:18:34.359348 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f37f8268fb0] request id: only_iris_and_face, model: face_02_landmark_detection_model, requested version: 1,
actual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3c3c008358] input: images, type: FP32, original shape: [1,3,640,480], batch + shape: [1,3,640,480], shape: [3,640,480]
override inputs:
inputs:
[0x0x7f3c3c008358] input: images, type: FP32, original shape: [1,3,640,480], batch + shape: [1,3,640,480], shape: [3,640,480]
original requested outputs:
output0
requested outputs:
output0

I0320 08:18:34.359391 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8000090
I0320 08:18:34.359403 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8087bf0
I0320 08:18:34.364608 810908 tensorrt.cc:381] model face_02_landmark_detection_model, instance face_02_landmark_detection_model_0, executing 1 requests
I0320 08:18:34.364661 810908 instance_state.cc:360] TRITONBACKEND_ModelExecute: Issuing face_02_landmark_detection_model_0 with 1 requests
I0320 08:18:34.364691 810908 instance_state.cc:409] TRITONBACKEND_ModelExecute: Running face_02_landmark_detection_model_0 with 1 requests
I0320 08:18:34.364754 810908 instance_state.cc:1437] Optimization profile default [0] is selected for face_02_landmark_detection_model_0
I0320 08:18:34.364866 810908 instance_state.cc:900] Context with profile default [0] is being executed for face_02_landmark_detection_model_0
I0320 08:18:34.367152 810908 infer_response.cc:167] add response output: output: output0, type: FP32, shape: [1,20,6300]
I0320 08:18:34.367207 810908 ensemble_scheduler.cc:555] Internal response allocation: output0, size 504000, addr 0x7f3ea8384000, memory type 2, type id 0
I0320 08:18:34.367830 810908 ensemble_scheduler.cc:570] Internal response release: size 504000, addr 0x7f3ea8384000
I0320 08:18:34.367914 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f37f8213270] request id: only_iris_and_face, model: face_03_preprocess, requested version: 1, actual version
: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3820004f78] input: RESP_JSON, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f3820004e78] input: FACE_TYPES, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f3820004d78] input: YOLO_MODEL_OUTPUT, type: FP32, original shape: [1,20,6300], batch + shape: [1,20,6300], shape: [1,20,6300]
[0x0x7f3e48009e28] input: PADDED_PROCESSED_IMAGES, type: FP32, original shape: [1,3,640,480], batch + shape: [1,3,640,480], shape: [1,3,640,480]
override inputs:
inputs:
[0x0x7f3e48009e28] input: PADDED_PROCESSED_IMAGES, type: FP32, original shape: [1,3,640,480], batch + shape: [1,3,640,480], shape: [1,3,640,480]
[0x0x7f3820004d78] input: YOLO_MODEL_OUTPUT, type: FP32, original shape: [1,20,6300], batch + shape: [1,20,6300], shape: [1,20,6300]
[0x0x7f3820004e78] input: FACE_TYPES, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f3820004f78] input: RESP_JSON, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
BEST_FACE_IMAGE_RGB
FACE_METADATAS
RESP_JSON
TRANSFORMATION_MATRIX
requested outputs:
BEST_FACE_IMAGE_RGB
FACE_METADATAS
RESP_JSON
TRANSFORMATION_MATRIX

I0320 08:18:34.368065 810908 instance_state.cc:1294] TRITONBACKEND_ModelExecute: model face_02_landmark_detection_model_0 released 1 requests
I0320 08:18:34.368144 810908 python_be.cc:1263] model face_03_preprocess, instance face_03_preprocess_0_9, executing 1 requests
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "face keypoint landmark in face attempt face_1 are [[225, 218, 295, 365, 359, 177, 304,
244, 195, 302]] with [[110, 65, 273, 382]] bbox having confidence score [0.846688985824585]"}
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "1 face detected"}
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "face landmark selected with confidence score 0.85 from face index face_1 of bounding bo
x [110, 65, 383, 447], landmark [[177, 225], [304, 218], [244, 295], [195, 365], [302, 359]]"}
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "tranformation matrix calculated successfully with value [[0.28288647532463074, -0.01437
4582096934319, -8.908162117004395], [0.014374582096934319, 0.28288647532463074, -14.328369140625]]"}
I0320 08:18:34.414954 810908 infer_response.cc:167] add response output: output: BEST_FACE_IMAGE_RGB, type: FP32, shape: [1,640,480,3]
I0320 08:18:34.415009 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 3686400, addr 0x7f3ee80c57c0
I0320 08:18:34.415040 810908 ensemble_scheduler.cc:555] Internal response allocation: BEST_FACE_IMAGE_RGB, size 3686400, addr 0x7f3ee80c57c0, memory type 1, type id 0
I0320 08:18:34.416327 810908 infer_response.cc:167] add response output: output: TRANSFORMATION_MATRIX, type: FP32, shape: [1,2,3]
I0320 08:18:34.416360 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 24, addr 0x7f3ee8087bf0
I0320 08:18:34.416381 810908 ensemble_scheduler.cc:555] Internal response allocation: TRANSFORMATION_MATRIX, size 24, addr 0x7f3ee8087bf0, memory type 1, type id 0
I0320 08:18:34.416405 810908 infer_response.cc:167] add response output: output: FACE_METADATAS, type: BYTES, shape: [1]
I0320 08:18:34.416427 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 51, addr 0x7f3ee8000090
I0320 08:18:34.416447 810908 ensemble_scheduler.cc:555] Internal response allocation: FACE_METADATAS, size 51, addr 0x7f3ee8000090, memory type 1, type id 0
I0320 08:18:34.416469 810908 infer_response.cc:167] add response output: output: RESP_JSON, type: BYTES, shape: [1]
I0320 08:18:34.416490 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 344, addr 0x7f3ee80000d0
I0320 08:18:34.416510 810908 ensemble_scheduler.cc:555] Internal response allocation: RESP_JSON, size 344, addr 0x7f3ee80000d0, memory type 1, type id 0
I0320 08:18:34.416549 810908 ensemble_scheduler.cc:570] Internal response release: size 3686400, addr 0x7f3ee80c57c0
I0320 08:18:34.416549 810908 ensemble_scheduler.cc:570] Internal response release: size 3686400, addr 0x7f3ee80c57c0                                                                                   [51/1858]
I0320 08:18:34.416570 810908 ensemble_scheduler.cc:570] Internal response release: size 24, addr 0x7f3ee8087bf0
I0320 08:18:34.416590 810908 ensemble_scheduler.cc:570] Internal response release: size 51, addr 0x7f3ee8000090
I0320 08:18:34.416610 810908 ensemble_scheduler.cc:570] Internal response release: size 344, addr 0x7f3ee80000d0
I0320 08:18:34.416664 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f3e15faefd0] request id: only_iris_and_face, model: face_04_model_preprocess, requested version: 1, actual v
ersion: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3e15faf478] input: TRANSFORMATION_MATRIX, type: FP32, original shape: [1,2,3], batch + shape: [1,2,3], shape: [2,3]
[0x0x7f3e15faf308] input: BEST_FACE_IMAGE_RGB, type: FP32, original shape: [1,640,480,3], batch + shape: [1,640,480,3], shape: [640,480,3]
override inputs:
inputs:
[0x0x7f3e15faf308] input: BEST_FACE_IMAGE_RGB, type: FP32, original shape: [1,640,480,3], batch + shape: [1,640,480,3], shape: [640,480,3]
[0x0x7f3e15faf478] input: TRANSFORMATION_MATRIX, type: FP32, original shape: [1,2,3], batch + shape: [1,2,3], shape: [2,3]
original requested outputs:
PROCESSED_IMAGES
requested outputs:
PROCESSED_IMAGES

I0320 08:18:34.416800 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8087c20
I0320 08:18:34.416832 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8087bc0
I0320 08:18:34.416869 810908 python_be.cc:2230] TRITONBACKEND_ModelInstanceExecute: model instance name face_03_preprocess_0_9 released 1 requests
I0320 08:18:34.428128 810908 infer_response.cc:167] add response output: output: PROCESSED_IMAGES, type: FP32, shape: [1,3,112,112]
I0320 08:18:34.428189 810908 ensemble_scheduler.cc:555] Internal response allocation: PROCESSED_IMAGES, size 150528, addr 0x7f3ea8000000, memory type 2, type id 0
I0320 08:18:34.428319 810908 ensemble_scheduler.cc:570] Internal response release: size 150528, addr 0x7f3ea8000000
I0320 08:18:34.428366 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f3d2000a270] request id: only_iris_and_face, model: face_05_embedding_model, requested version: 1, actual ve
rsion: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f3d2000a5a8] input: data, type: FP32, original shape: [1,3,112,112], batch + shape: [1,3,112,112], shape: [3,112,112]
override inputs:
inputs:
[0x0x7f3d2000a5a8] input: data, type: FP32, original shape: [1,3,112,112], batch + shape: [1,3,112,112], shape: [3,112,112]
original requested outputs:
fc1
requested outputs:
fc1

I0320 08:18:34.428457 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8087bf0
I0320 08:18:34.428491 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee80c57c0
I0320 08:18:34.433683 810908 tensorrt.cc:381] model face_05_embedding_model, instance face_05_embedding_model_0, executing 1 requests
I0320 08:18:34.433727 810908 instance_state.cc:360] TRITONBACKEND_ModelExecute: Issuing face_05_embedding_model_0 with 1 requests
I0320 08:18:34.433749 810908 instance_state.cc:409] TRITONBACKEND_ModelExecute: Running face_05_embedding_model_0 with 1 requests
I0320 08:18:34.433802 810908 instance_state.cc:1437] Optimization profile default [0] is selected for face_05_embedding_model_0
I0320 08:18:34.433900 810908 instance_state.cc:900] Context with profile default [0] is being executed for face_05_embedding_model_0
I0320 08:18:34.434802 810908 infer_response.cc:167] add response output: output: fc1, type: FP32, shape: [1,512]
I0320 08:18:34.434848 810908 ensemble_scheduler.cc:555] Internal response allocation: fc1, size 2048, addr 0x7f3ea8024c00, memory type 2, type id 0
I0320 08:18:34.438130 810908 ensemble_scheduler.cc:570] Internal response release: size 2048, addr 0x7f3ea8024c00
I0320 08:18:34.438211 810908 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7f3818007f90] request id: only_iris_and_face, model: face_06_embedding_postprocess, requested version: 1, act
ual version: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f382c004168] input: RESP_JSON, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f382c004068] input: EMBEDDINGS, type: FP32, original shape: [1,512], batch + shape: [1,512], shape: [1,512]
[0x0x7f38180082c8] input: FACE_METADATAS, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
override inputs:
inputs:
[0x0x7f38180082c8] input: FACE_METADATAS, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f382c004068] input: EMBEDDINGS, type: FP32, original shape: [1,512], batch + shape: [1,512], shape: [1,512]
[0x0x7f382c004168] input: RESP_JSON, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f38180082c8] input: FACE_METADATAS, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
[0x0x7f382c004068] input: EMBEDDINGS, type: FP32, original shape: [1,512], batch + shape: [1,512], shape: [1,512]
[0x0x7f382c004168] input: RESP_JSON, type: BYTES, original shape: [1], batch + shape: [1], shape: [1]
original requested outputs:
RESP_DATA
requested outputs:
RESP_DATA

I0320 08:18:34.438342 810908 instance_state.cc:1294] TRITONBACKEND_ModelExecute: model face_05_embedding_model_0 released 1 requests
I0320 08:18:34.438493 810908 python_be.cc:1263] model face_06_embedding_postprocess, instance face_06_embedding_postprocess_0_0, executing 1 requests
{"timestamp": "2026-03-20T08:18:34+0000", "code": "102", "txnRefId": "bbc1d5cc-f800-4898-abe8-1f37e5b34eb5", "message": "face embedding generated successfully", "response_time_in_ms": 139}
I0320 08:18:34.464309 810908 infer_response.cc:167] add response output: output: RESP_DATA, type: BYTES, shape: [1]
I0320 08:18:34.464366 810908 pinned_memory_manager.cc:162] pinned memory allocation: size 2914, addr 0x7f3ee8000230
I0320 08:18:34.464390 810908 ensemble_scheduler.cc:555] Internal response allocation: RESP_DATA, size 2914, addr 0x7f3ee8000230, memory type 1, type id 0
I0320 08:18:34.464424 810908 ensemble_scheduler.cc:570] Internal response release: size 2914, addr 0x7f3ee8000230
I0320 08:18:34.464612 810908 infer_response.cc:141] add response output: output: RESP_DATA, type: BYTES, shape: [1]
I0320 08:18:34.464646 810908 http_server.cc:1103] HTTP: unable to provide 'RESP_DATA' in CPU_PINNED, will use CPU
I0320 08:18:34.464693 810908 http_server.cc:1123] HTTP using buffer for: 'RESP_DATA', size: 2914, addr: 0x7f3b0c005610
I0320 08:18:34.464726 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8000230
I0320 08:18:34.465407 810908 http_server.cc:1197] HTTP release: size 2914, addr 0x7f3b0c005610
I0320 08:18:34.465511 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee80000d0
I0320 08:18:34.465549 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8000090
I0320 08:18:34.465591 810908 python_be.cc:2230] TRITONBACKEND_ModelInstanceExecute: model instance name face_06_embedding_postprocess_0_0 released 1 requests
E0320 08:18:39.365122 810908 dali_model_instance.cc:46] Critical error in pipeline:
Error when executing Mixed operator decoders__Image encountered:
Error in thread 0: [/opt/dali/dali/operators/decoder/nvjpeg/nvjpeg_helper.h:233] [/opt/dali/dali/image/image_factory.cc:100] Unrecognized image format. Supported formats are: JPEG, PNG, BMP, TIFF, PNM, JPEG20
00 and WebP.
Stacktrace (9 entries):
[frame 0]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0xc32fb) [0x7f3ea57f22fb]
[frame 1]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x8a604) [0x7f3ea57b9604]
[frame 2]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7544d) [0x7f3e7ddff44d]
[frame 3]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7b725) [0x7f3e7de05725]
[frame 4]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7e1a4) [0x7f3e7de081a4]
[frame 5]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(dali::ThreadPool::ThreadMain(int, int, bool, std::string const&)+0x1e6) [0x7f3ea58d86c6]
[frame 6]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x6baa80) [0x7f3ea5de9a80]
[frame 7]: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f404040eb43]
[frame 8]: /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f404049fbb4]
. File:
Stacktrace (8 entries):
[frame 0]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0x67183e) [0x7f3e7d9fb83e]
[frame 1]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa75804) [0x7f3e7ddff804]
[frame 2]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7b725) [0x7f3e7de05725]
[frame 3]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali_operators.so(+0xa7e1a4) [0x7f3e7de081a4]
[frame 4]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(dali::ThreadPool::ThreadMain(int, int, bool, std::string const&)+0x1e6) [0x7f3ea58d86c6]
[frame 5]: /opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.10/site-packages/nvidia/dali/libdali.so(+0x6baa80) [0x7f3ea5de9a80]
[frame 6]: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f404040eb43]
[frame 7]: /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f404049fbb4]

Current pipeline object is no longer valid.
I0320 08:18:39.365249 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee80c55e0
I0320 08:18:39.365267 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee80c5640
I0320 08:18:39.365289 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee8087d80
I0320 08:18:39.365301 810908 pinned_memory_manager.cc:191] pinned memory deallocation: addr 0x7f3ee80c5610

setting the thread to 1 in dali; still produces the same output.
Full Log out (1).txt

[banasraf]

Thanks for providing the info!
From what I see in the logs, the requests were not batched togeterh, they are processed separately:

I0320 08:25:44.733958 820298 infer_request.cc:751] [request id: decoding_failed] prepared: [0x0x7ff66c0068c0] request id: decoding_failed, model: face_01_landmark_detection_preprocess, requested version: 1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7ff66c006d68] input: INPUT_BUFFERS, type: UINT8, original shape: [1,251977], batch + shape: [1,251977], shape: [251977]
[0x0x7ff66c006bf8] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
override inputs:
inputs:
[0x0x7ff66c006bf8] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
[0x0x7ff66c006d68] input: INPUT_BUFFERS, type: UINT8, original shape: [1,251977], batch + shape: [1,251977], shape: [251977]
original requested outputs:
PADDED_PROCESSED_IMAGES
requested outputs:
PADDED_PROCESSED_IMAGES

I0320 08:25:44.747051 820298 infer_request.cc:751] [request id: only_iris_and_face] prepared: [0x0x7ff7a00068d0] request id: only_iris_and_face, model: face_01_landmark_detection_preprocess, requested version: 1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7ff7a0006d78] input: INPUT_BUFFERS, type: UINT8, original shape: [1,555805], batch + shape: [1,555805], shape: [555805]
[0x0x7ff7a0006c08] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
override inputs:
inputs:
[0x0x7ff7a0006c08] input: BUFFER_SIZES, type: UINT32, original shape: [1,1], batch + shape: [1,1], shape: [1]
[0x0x7ff7a0006d78] input: INPUT_BUFFERS, type: UINT8, original shape: [1,555805], batch + shape: [1,555805], shape: [555805]
original requested outputs:
PADDED_PROCESSED_IMAGES
requested outputs:
PADDED_PROCESSED_IMAGES

I am not sure why is that. Can you tell me why are you using the preserve_ordering: true parameter for the DALI model and if the behaviour is the same if you do not add it?

[lokesh-kurre]

Indeed I noticed, request are not getting batched.
But as per my knowledge of dynamic batching schedular, the request should be batched unless resultant batch size is more than max batch size of next step (here dali) or max delay has been elapsed.
So, here in my case, resultant batch size is of 2 & delay has been set a sufficiently large value. Therefore, batching should happen.

As for preserve_ordering: true is used as safety net, to ensure output doesn't become out of order compare to inputs. However, exact use case is not sure, if it impact on performace or not.

The preserve_ordering property is used to force all responses to be returned in the same order as requests were received. See the protobuf documentation for details. Source