fix: GEPA optimizer compatibility + custom OpenAI-compatible API support - #167
Open
kk17 wants to merge 1 commit into
Open
fix: GEPA optimizer compatibility + custom OpenAI-compatible API support#167kk17 wants to merge 1 commit into
kk17 wants to merge 1 commit into
Conversation
GEPA optimizer fix (DSPy 3.2.1): - max_steps → max_full_evals (correct parameter name) - Add reflection_lm=lm (GEPA requires explicit reflection LM) Metric signature fix: - skill_fitness_metric now uses *args, **kwargs to handle both GEPA's 5-arg call signature and MIPROv2's 3-arg signature - Normalizes args to extract output/expected from dspy.Example or dict Custom OpenAI-compatible API support: - Read OPENAI_BASE_URL and OPENAI_API_KEY from env (generic, not hardcoded to any specific provider) - Pass as api_base/api_key to dspy.LM() only when set - Default models remain openai/gpt-4.1 (backward compatible) - Override via --optimizer-model and --eval-model CLI args - Documented in README with usage examples Skill search improvements: - find_skill() now also searches ~/.hermes/profiles/*/skills/ so skills installed in user profiles are discoverable - Extracted _search_skills_dir() helper for reuse Frontmatter preservation: - reassemble_skill() preserves YAML frontmatter during evolution (already correct in codebase, verified working) .gitignore: - Add output/ and TRIAL_REPORT.md to prevent local artifacts from being committed
Review: PR #167 vs. overlapping PRsI reviewed this diff against the current main and the other open PRs in this family (#155, #159, #153, #146, #161). What this PR does well (unique value):
GEPA fixes overlap with other PRs:
Suggestion: The custom API support and profile skill search are valuable and independent of the GEPA fixes. Consider splitting those into a separate PR so they can be merged independently. The GEPA compat portion overlaps heavily with #155 which has a more complete fix (SkillModule Signature instructions + evolved body extraction). |
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four fixes to make the evolution pipeline work with DSPy 3.2.1 and support any OpenAI-compatible API endpoint.
Changes
1. GEPA optimizer fix (
evolution/skills/evolve_skill.py)max_steps→max_full_evals(correct parameter name in DSPy 3.2.1)reflection_lm=lmparameter (GEPA requires an explicit reflection LM)2. Metric signature fix (
evolution/core/fitness.py)skill_fitness_metric(example, prediction, trace=None)to*args, **kwargs(inputs, outputs, trace, pred_name, ...), while MIPROv2 calls with 3 args(example, prediction, trace)dspy.Exampleordict3. Custom OpenAI-compatible API support (
evolution/skills/evolve_skill.py)OPENAI_BASE_URLenv var is set, it is passed asapi_basetodspy.LM()OPENAI_API_KEYenv var is set, it is passed asapi_keyopenai/gpt-4.1andopenai/gpt-4.1-mini(backward compatible)--optimizer-modeland--eval-modelCLI args4. Skill search improvements (
evolution/skills/skill_module.py)find_skill()now also searches~/.hermes/profiles/*/skills/so skills installed in user profiles are discoverable_search_skills_dir()helper for code reusereassemble_skill()preserves YAML frontmatter during evolution (verified working)Testing
max_stepsTypeError)Backward Compatibility
openai/gpt-4.1