fix: make the GEPA skill-evolution path work end-to-end - #168
Open
enzo-adami wants to merge 5 commits into
Open
Conversation
…lve it
GEPA's candidate space is {name: pred.signature.instructions} over
named_predictors() (dspy/teleprompt/gepa/gepa.py:126,599). SkillModule kept
the skill body in a plain attribute passed as an InputField, which GEPA
never mutates — optimized_module.skill_text always returned the original
text, so the saved evolved_skill.md could never contain the evolution.
Move the skill body into the predictor's signature instructions via
with_instructions(), and make skill_text a read-through property over
named_predictors() so extraction reflects the evolved candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
validate_all() was called with skill['body'] / evolved_body, but
_check_skill_structure() looks for YAML frontmatter — which load_skill()
strips from body. The structure constraint therefore failed on every run
and no evolved candidate could ever pass the gate ('not deploying').
Validate skill['raw'] for the baseline and the reassembled evolved_full
for the candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y runs dspy.GEPA has no `max_steps` parameter — the previous call raised TypeError on every run and silently fell back to MIPROv2, so the GEPA path advertised by the README could never execute. GEPA also requires exactly one budget parameter and a reflection LM, and its GEPAFeedbackMetric protocol calls the metric with (gold, pred, trace, pred_name, pred_trace). - Pass max_full_evals=iterations and reflection_lm (optimizer model). - Extend skill_fitness_metric to the GEPA calling convention; return Prediction(score, feedback) with deterministic missing-terms feedback for predictor-level reflection calls, plain float everywhere else (holdout and MIPROv2 behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reasoning models served by local OpenAI-compatible endpoints often sit
behind low server-side max_tokens defaults: the thinking phase consumes
the whole budget and content comes back empty ('LM returned an empty or
null response' on every rollout). Let callers set an explicit generation
budget and temperature for both the eval LM and GEPA's reflection LM.
Defaults unchanged (kwargs only passed when provided).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…al local endpoints On a serial local endpoint (decode-concurrency 1), GEPA's parallel rollouts queue behind each other; once queue depth times per-request latency exceeds the client timeout, every in-flight call times out in cascade and litellm's default retries re-pile the queue (observed: 'Average Metric: 0.00 / 0' with APITimeoutError on every rollout, while the server kept generating for closed sockets). Let callers serialize evaluation (GEPA num_threads) and bound per-request timeout/retries. Defaults unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four commits, each removing a blocker that prevented Phase 1 (skill evolution) from ever producing a validated evolved skill. Exercised end-to-end with dspy 3.3.0 / gepa 0.1.1 against a local OpenAI-compatible endpoint: GEPA now actually runs (no silent fallback) and evolved candidates can pass the constraint gate.
Bug 1 — evolved text never reached the artifact (5915499)
GEPA's candidate space is exactly
{name: pred.signature.instructions}overnamed_predictors()(dspyteleprompt/gepa/gepa.py, seed + candidate build).SkillModulekept the skill body in a plain attribute passed as anInputField, which GEPA never mutates — sooptimized_module.skill_textalways returned the original text andevolved_skill.mdwas always identical to the baseline. The optimizer measured improvements it never persisted.Fix: the skill body becomes the predictor's signature instructions (
with_instructions());skill_textis a read-through property overnamed_predictors(), so extraction reflects the evolved candidate with no change to the extraction call site.Bug 2 — constraint gate rejected every candidate (44caaf5)
validate_all()was called withskill['body']/evolved_body, but_check_skill_structure()looks for YAML frontmatter — whichload_skill()strips frombody. The structure constraint therefore failed on every run and every evolved candidate ended in 'not deploying'.Fix: validate
skill['raw']for the baseline and the reassembledevolved_fullfor the candidate.Bug 3 — GEPA never ran at all (916661d)
dspy.GEPAhas nomax_stepsparameter: the call raisedTypeErroron every run and silently fell back to MIPROv2, so the GEPA path advertised in the README could never execute. GEPA also requires exactly one budget parameter and a reflection LM, and itsGEPAFeedbackMetricprotocol calls the metric with(gold, pred, trace, pred_name, pred_trace).Fix: pass
max_full_evals=iterationsandreflection_lm; extendskill_fitness_metricto the GEPA calling convention, returningPrediction(score, feedback)with deterministic missing-terms feedback for predictor-level reflection calls and a plain float everywhere else (holdout and MIPROv2 behavior unchanged).Enhancement — generation budget for local endpoints (5b7fbd9)
Reasoning models served by local OpenAI-compatible endpoints often sit behind low server-side
max_tokensdefaults: the thinking phase consumes the whole budget and content comes back empty ('LM returned an empty or null response' on every rollout). New--max-tokens/--temperatureoptions apply to both the eval LM and GEPA's reflection LM; defaults unchanged.Tests
145 → 154 (9 new: GEPA-visibility of the skill text, full-file vs bare-body constraint regression, metric contract incl. the
max_stepsregression), all green.🤖 Generated with Claude Code