Skip to content

Fix vLLM wrapper option forwarding - #2298

Open
patrick-andstar wants to merge 1 commit into
QwenLM:mainfrom
patrick-andstar:fix/vllm-wrapper-options
Open

Fix vLLM wrapper option forwarding#2298
patrick-andstar wants to merge 1 commit into
QwenLM:mainfrom
patrick-andstar:fix/vllm-wrapper-options

Conversation

@patrick-andstar

Copy link
Copy Markdown

Summary

  • Forward the caller's quantization option to vLLM.
  • Include caller-provided stop token IDs in vLLM sampling parameters.
  • Store (query, response) pairs in returned history.

Motivation

The vLLM wrapper silently ignored supported call options and persisted the generated prompt instead of the user query, producing incorrect follow-up context. The same implementation exists in the example and recipe copies.

Testing

  • Ran a stubbed vLLM regression harness against both wrapper copies.
  • Ran AST syntax checks for both files.
  • Ran git diff --check.
  • Did not run real vLLM/model integration because it requires the vLLM runtime and a checkpoint.

Duplicate Work Check

Searched open Issues and PRs for vllm_wrapper, quantization, and stop_words_ids; no matching open change was found. Closed issue #975 reported related vLLM context behavior but did not change this implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant