Skip to content

Introduce support for dynamic batching#1662

Merged
baijumeswani merged 5 commits into
mainfrom
baijumeswani/dynamic-batching
Sep 26, 2025
Merged

Introduce support for dynamic batching#1662
baijumeswani merged 5 commits into
mainfrom
baijumeswani/dynamic-batching

Conversation

@baijumeswani

@baijumeswani baijumeswani commented Jul 29, 2025

Copy link
Copy Markdown
Collaborator

The changes in #1580 introduce a new API in onnxruntime-genai that enables continuous submission of requests to a generation engine. Internally, the scheduler statically batches these requests, allowing the API to interface seamlessly with models designed for static batching.

This pull request lays the groundwork for dynamic request batching by incorporating concepts from vLLM. Specifically, it implements a block-based key-value cache manager.

To support dynamic batching, the underlying model must handle variable-length inputs (as opposed to fixed-size static batches) and leverage the PagedAttention contrib-op.

To run the continuous batching example:

python continuous-batching.py -m models/phi-3.5-paged/ -e cuda -n 100
100%|███████████████████████████████████████████████████████████████████████| 100/100 [00:09<00:00, 11.07it/s]
⌛ Tokens per second: 2630.78

Base automatically changed from baijumeswani/engine to main August 4, 2025 16:23
@baijumeswani baijumeswani force-pushed the baijumeswani/dynamic-batching branch 3 times, most recently from 974151e to 03ed710 Compare September 9, 2025 17:16
@baijumeswani baijumeswani marked this pull request as ready for review September 9, 2025 17:17
Comment thread src/config.h Outdated
Comment thread src/config.h
Comment thread src/engine/block.cpp
Comment thread src/engine/block.cpp Outdated
Comment thread src/engine/decoders/static_batch_decoder_io.h Outdated
Comment thread src/engine/decoders/varlen_decoder_io.h
Comment thread src/engine/paged_key_value_cache.cpp Outdated
Comment thread src/smartptrs.h Outdated
@baijumeswani baijumeswani force-pushed the baijumeswani/dynamic-batching branch from 03ed710 to dc284da Compare September 25, 2025 22:30
@baijumeswani

Copy link
Copy Markdown
Collaborator Author

Thank you for the review. :)

@baijumeswani baijumeswani enabled auto-merge (squash) September 25, 2025 22:43
@baijumeswani baijumeswani merged commit 562091d into main Sep 26, 2025
15 checks passed
@baijumeswani baijumeswani deleted the baijumeswani/dynamic-batching branch September 26, 2025 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants