feat: first-class vLLM / Hugging Face local-model support (#359) - #413
Open
LakshyAAAgrawal wants to merge 1 commit into
Open
LakshyAAAgrawal wants to merge 1 commit into
LakshyAAAgrawal wants to merge 1 commit into
Conversation
Add an optional, opt-in path for running GEPA against local open-weight models served by vLLM (or any OpenAI-compatible endpoint), without touching core behavior or the dependency-free core. - `make_vllm_lm(model, api_base=..., api_key="EMPTY", ...)` in `gepa.lm`: a thin convenience wrapper that auto-prefixes the `openai/` provider and returns an `LM`. Usable as both the task and reflection model. Adds no import-time dependency (litellm stays lazy); GEPA reaches the server over HTTP and never imports `vllm`. - Re-exported from `gepa.optimize_anything` next to `make_litellm_lm` for discoverability. Deliberately NOT surfaced at top-level `gepa` — this is an added feature, not the default route. - Optional `vllm` extra: intentionally lean (`gepa[full]`, the LiteLLM client used to call the endpoint). The heavy, GPU/platform-specific `vllm` server is installed separately on the serving box (`pip install vllm`), so no torch/CUDA footprint is added to the core or the lockfile. - Docs: new guide `guides/local-models-vllm.md` (serve, call, optimize, mixed local/hosted setups, cost/token notes, troubleshooting) + nav entry. - Runnable example under `examples/vllm_huggingface/` (README + main.py). - Tests: `tests/test_vllm_lm.py` — 12 tests covering prefixing, defaults, and api_base/api_key/sampling forwarding. litellm is mocked; no server or model download required. Closes #359 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #359.
Adds an optional, opt-in path for running GEPA against local open-weight models served by vLLM (or any OpenAI-compatible endpoint). Core GEPA stays dependency-free and no default changes.
What's here
gepa.lm.make_vllm_lm(model, api_base="http://localhost:8000/v1", api_key="EMPTY", ...)— a thin convenience wrapper that auto-prefixes theopenai/provider and returns anLM. Works as both the task and reflection model. Adds no import-time dependency (litellm stays lazy); GEPA reaches the server over HTTP and never importsvllmin-process.gepa.optimize_anythingnext tomake_litellm_lmfor discoverability. Deliberately not surfaced at top-levelgepa— this is an added feature, not the default route.guides/local-models-vllm.md(serve → call → optimize, mixed local/hosted setups, cost/token notes, troubleshooting) + mkdocs nav entry.examples/vllm_huggingface/(README +main.py).tests/test_vllm_lm.py— 12 tests covering provider prefixing, defaults, andapi_base/api_key/sampling forwarding. litellm is mocked; no server or model download.On the
vllmextra (design note)The optional
vllmextra is intentionally lean (gepa[full]— the LiteLLM client used to call the endpoint). The heavy, GPU/platform-specific vLLM server is installed separately on the serving box (pip install vllm), matching the architecture (GEPA talks to it over HTTP). This keeps zero torch/CUDA footprint out of the core and the lockfile — theuv.lockchange is a single flattened extra entry, not the torch/CUDA/ray universe. Happy to bundle the server into the extra instead if you'd prefer the one-command install; it's a one-line change (at the cost of ~1.5k lines of heavy lockfile churn).Verification
uv lock --check✅ (lock consistent for CI--locked)ruff check✅pyrightonsrc/✅ and on the example ✅pytest tests/test_vllm_lm.py✅ 12 passed; existing lm/optimize_anything tests ✅ 115 passed / 4 skipped🤖 Generated with Claude Code