Context
HybridEngineRollout supports optional profiling through
HybridEngineRolloutConfig.enable_profiling.
When continuous batching is enabled through
SamplingConfig.continuous_batch_size, the continuous path currently bypasses
the profiling logic. As a result, generate() completes without recording a
profiling snapshot, and get_last_profile() does not provide useful timing
information for continuous-batching workloads.
Proposed work
Define and implement profiling support for continuous batching.
The profiling contract should clarify how to measure and report:
- total rollout time;
- prompt prefill time;
- decode time;
- scheduler and cache-management overhead;
- number of decode forwards;
- generated tokens and throughput;
- active batch size and continuous-batching capacity.
Acceptance criteria
enable_profiling=True produces a profiling snapshot for continuous
batching.
- The reported fields have documented meanings for variable active batch sizes
and independently retired rows.
- Profiling remains disabled by default.
- Enabling profiling does not change generated outputs or scheduling behavior.
- Tests cover the profiling output for a deterministic continuous-batching
workload.
Follow-up from DeepSpeed PR #8368. cc @delock
Context
HybridEngineRolloutsupports optional profiling throughHybridEngineRolloutConfig.enable_profiling.When continuous batching is enabled through
SamplingConfig.continuous_batch_size, the continuous path currently bypassesthe profiling logic. As a result,
generate()completes without recording aprofiling snapshot, and
get_last_profile()does not provide useful timinginformation for continuous-batching workloads.
Proposed work
Define and implement profiling support for continuous batching.
The profiling contract should clarify how to measure and report:
Acceptance criteria
enable_profiling=Trueproduces a profiling snapshot for continuousbatching.
and independently retired rows.
workload.
Follow-up from DeepSpeed PR #8368. cc @delock