Skip to content

fix: GEPA optimizer compatibility + custom OpenAI-compatible API support - #167

Open
kk17 wants to merge 1 commit into
NousResearch:mainfrom
kk17:fix/gepa-and-custom-api-support
Open

fix: GEPA optimizer compatibility + custom OpenAI-compatible API support#167
kk17 wants to merge 1 commit into
NousResearch:mainfrom
kk17:fix/gepa-and-custom-api-support

Conversation

@kk17

@kk17 kk17 commented Aug 3, 2026

Copy link
Copy Markdown

Summary

Four fixes to make the evolution pipeline work with DSPy 3.2.1 and support any OpenAI-compatible API endpoint.

Changes

1. GEPA optimizer fix (evolution/skills/evolve_skill.py)

  • max_stepsmax_full_evals (correct parameter name in DSPy 3.2.1)
  • Add reflection_lm=lm parameter (GEPA requires an explicit reflection LM)

2. Metric signature fix (evolution/core/fitness.py)

  • Changed skill_fitness_metric(example, prediction, trace=None) to *args, **kwargs
  • GEPA calls the metric with 5 args (inputs, outputs, trace, pred_name, ...), while MIPROv2 calls with 3 args (example, prediction, trace)
  • The flexible signature normalizes both call patterns, extracting output/expected from dspy.Example or dict

3. Custom OpenAI-compatible API support (evolution/skills/evolve_skill.py)

  • If OPENAI_BASE_URL env var is set, it is passed as api_base to dspy.LM()
  • If OPENAI_API_KEY env var is set, it is passed as api_key
  • Both are optional — if unset, DSPy defaults are used
  • Default models remain openai/gpt-4.1 and openai/gpt-4.1-mini (backward compatible)
  • Users can override via --optimizer-model and --eval-model CLI args
  • This enables OmniRoute, vLLM, LiteLLM, Ollama, and any other OpenAI-compatible API
  • Documented in README with usage examples

4. Skill search improvements (evolution/skills/skill_module.py)

  • find_skill() now also searches ~/.hermes/profiles/*/skills/ so skills installed in user profiles are discoverable
  • Extracted _search_skills_dir() helper for code reuse
  • reassemble_skill() preserves YAML frontmatter during evolution (verified working)

Testing

  • Validated with a real evolution run using an OpenAI-compatible API endpoint (OmniRoute)
  • GEPA optimizer now starts correctly (previously crashed with max_steps TypeError)
  • Metric function handles both GEPA and MIPROv2 call signatures
  • Frontmatter preserved in evolved skill output

Backward Compatibility

  • All changes are backward compatible — default behavior is unchanged
  • No hardcoded URLs or API keys — everything is env-var driven
  • Default models remain openai/gpt-4.1

GEPA optimizer fix (DSPy 3.2.1):
- max_steps → max_full_evals (correct parameter name)
- Add reflection_lm=lm (GEPA requires explicit reflection LM)

Metric signature fix:
- skill_fitness_metric now uses *args, **kwargs to handle both
  GEPA's 5-arg call signature and MIPROv2's 3-arg signature
- Normalizes args to extract output/expected from dspy.Example or dict

Custom OpenAI-compatible API support:
- Read OPENAI_BASE_URL and OPENAI_API_KEY from env (generic, not
  hardcoded to any specific provider)
- Pass as api_base/api_key to dspy.LM() only when set
- Default models remain openai/gpt-4.1 (backward compatible)
- Override via --optimizer-model and --eval-model CLI args
- Documented in README with usage examples

Skill search improvements:
- find_skill() now also searches ~/.hermes/profiles/*/skills/
  so skills installed in user profiles are discoverable
- Extracted _search_skills_dir() helper for reuse

Frontmatter preservation:
- reassemble_skill() preserves YAML frontmatter during evolution
  (already correct in codebase, verified working)

.gitignore:
- Add output/ and TRIAL_REPORT.md to prevent local artifacts
  from being committed
@TurtleMcTurtle

Copy link
Copy Markdown

Review: PR #167 vs. overlapping PRs

I reviewed this diff against the current main and the other open PRs in this family (#155, #159, #153, #146, #161).

What this PR does well (unique value):

  • ✅ Custom OpenAI-compatible API support via OPENAI_BASE_URL/OPENAI_API_KEY env vars — this is not addressed by any other PR and is genuinely useful for OmniRoute/vLLM/LiteLLM/Ollama users
  • find_skill() refactored to search Hermes profile skills directories — also unique to this PR
  • ✅ README documentation for custom API usage

GEPA fixes overlap with other PRs:

Suggestion: The custom API support and profile skill search are valuable and independent of the GEPA fixes. Consider splitting those into a separate PR so they can be merged independently. The GEPA compat portion overlaps heavily with #155 which has a more complete fix (SkillModule Signature instructions + evolved body extraction).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants