Skip to content

[BUG] Do not silently ignore use_shared_prefill with continuous batching #8458

Description

@nathon-lee

Context

HybridEngineRollout supports both shared prompt prefill and continuous
batching, but their interaction is currently undefined.

When continuous batching is enabled through
SamplingConfig.continuous_batch_size, the continuous path is selected before
the shared-prefill logic. As a result, HybridEngineRolloutConfig.use_shared_prefill
is silently ignored.

Continuous batching currently requires n_samples_per_prompt=1, so shared
prefill does not provide a clear benefit in this mode. However, silently
ignoring the configuration can be confusing and may hide an invalid combination
of options.

Proposed work

Define and document the behavior of use_shared_prefill when continuous
batching is enabled.

Possible choices include:

  • explicitly reject the combination with a clear ValueError; or
  • define and implement compatible shared-prefill semantics for continuous
    batching.

Acceptance criteria

  • The interaction between use_shared_prefill and continuous batching is
    explicitly defined.
  • The configuration is not silently ignored.
  • Tests cover both compatible and incompatible configurations.
  • The default non-continuous generation path remains unchanged.

Follow-up from DeepSpeed PR #8368. cc @delock

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtraining

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions