Skip to content

feat: first-class vLLM / Hugging Face local-model support (#359) - #413

Open
LakshyAAAgrawal wants to merge 1 commit into
mainfrom
feat/vllm-huggingface-support
Open

LakshyAAAgrawal wants to merge 1 commit into
mainfrom
feat/vllm-huggingface-support

Conversation

@LakshyAAAgrawal

Copy link
Copy Markdown
Contributor

Closes #359.

Adds an optional, opt-in path for running GEPA against local open-weight models served by vLLM (or any OpenAI-compatible endpoint). Core GEPA stays dependency-free and no default changes.

What's here

  • gepa.lm.make_vllm_lm(model, api_base="http://localhost:8000/v1", api_key="EMPTY", ...) — a thin convenience wrapper that auto-prefixes the openai/ provider and returns an LM. Works as both the task and reflection model. Adds no import-time dependency (litellm stays lazy); GEPA reaches the server over HTTP and never imports vllm in-process.
  • Re-exported from gepa.optimize_anything next to make_litellm_lm for discoverability. Deliberately not surfaced at top-level gepa — this is an added feature, not the default route.
  • Docs: new guide guides/local-models-vllm.md (serve → call → optimize, mixed local/hosted setups, cost/token notes, troubleshooting) + mkdocs nav entry.
  • Example: runnable examples/vllm_huggingface/ (README + main.py).
  • Tests: tests/test_vllm_lm.py — 12 tests covering provider prefixing, defaults, and api_base/api_key/sampling forwarding. litellm is mocked; no server or model download.

On the vllm extra (design note)

The optional vllm extra is intentionally lean (gepa[full] — the LiteLLM client used to call the endpoint). The heavy, GPU/platform-specific vLLM server is installed separately on the serving box (pip install vllm), matching the architecture (GEPA talks to it over HTTP). This keeps zero torch/CUDA footprint out of the core and the lockfile — the uv.lock change is a single flattened extra entry, not the torch/CUDA/ray universe. Happy to bundle the server into the extra instead if you'd prefer the one-command install; it's a one-line change (at the cost of ~1.5k lines of heavy lockfile churn).

Verification

  • uv lock --check ✅ (lock consistent for CI --locked)
  • ruff check
  • pyright on src/ ✅ and on the example ✅
  • pytest tests/test_vllm_lm.py ✅ 12 passed; existing lm/optimize_anything tests ✅ 115 passed / 4 skipped

🤖 Generated with Claude Code

Add an optional, opt-in path for running GEPA against local open-weight
models served by vLLM (or any OpenAI-compatible endpoint), without touching
core behavior or the dependency-free core.

- `make_vllm_lm(model, api_base=..., api_key="EMPTY", ...)` in `gepa.lm`:
  a thin convenience wrapper that auto-prefixes the `openai/` provider and
  returns an `LM`. Usable as both the task and reflection model. Adds no
  import-time dependency (litellm stays lazy); GEPA reaches the server over
  HTTP and never imports `vllm`.
- Re-exported from `gepa.optimize_anything` next to `make_litellm_lm` for
  discoverability. Deliberately NOT surfaced at top-level `gepa` — this is an
  added feature, not the default route.
- Optional `vllm` extra: intentionally lean (`gepa[full]`, the LiteLLM client
  used to call the endpoint). The heavy, GPU/platform-specific `vllm` server
  is installed separately on the serving box (`pip install vllm`), so no
  torch/CUDA footprint is added to the core or the lockfile.
- Docs: new guide `guides/local-models-vllm.md` (serve, call, optimize, mixed
  local/hosted setups, cost/token notes, troubleshooting) + nav entry.
- Runnable example under `examples/vllm_huggingface/` (README + main.py).
- Tests: `tests/test_vllm_lm.py` — 12 tests covering prefixing, defaults, and
  api_base/api_key/sampling forwarding. litellm is mocked; no server or model
  download required.

Closes #359

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add first-class vLLM / Hugging Face model loading support

1 participant