Skip to content

fix: make the GEPA skill-evolution path work end-to-end - #168

Open
enzo-adami wants to merge 5 commits into
NousResearch:mainfrom
enzo-adami:fix/gepa-skill-evolution-path
Open

fix: make the GEPA skill-evolution path work end-to-end#168
enzo-adami wants to merge 5 commits into
NousResearch:mainfrom
enzo-adami:fix/gepa-skill-evolution-path

Conversation

@enzo-adami

Copy link
Copy Markdown

Summary

Four commits, each removing a blocker that prevented Phase 1 (skill evolution) from ever producing a validated evolved skill. Exercised end-to-end with dspy 3.3.0 / gepa 0.1.1 against a local OpenAI-compatible endpoint: GEPA now actually runs (no silent fallback) and evolved candidates can pass the constraint gate.

Bug 1 — evolved text never reached the artifact (5915499)

GEPA's candidate space is exactly {name: pred.signature.instructions} over named_predictors() (dspy teleprompt/gepa/gepa.py, seed + candidate build). SkillModule kept the skill body in a plain attribute passed as an InputField, which GEPA never mutates — so optimized_module.skill_text always returned the original text and evolved_skill.md was always identical to the baseline. The optimizer measured improvements it never persisted.

Fix: the skill body becomes the predictor's signature instructions (with_instructions()); skill_text is a read-through property over named_predictors(), so extraction reflects the evolved candidate with no change to the extraction call site.

Bug 2 — constraint gate rejected every candidate (44caaf5)

validate_all() was called with skill['body'] / evolved_body, but _check_skill_structure() looks for YAML frontmatter — which load_skill() strips from body. The structure constraint therefore failed on every run and every evolved candidate ended in 'not deploying'.

Fix: validate skill['raw'] for the baseline and the reassembled evolved_full for the candidate.

Bug 3 — GEPA never ran at all (916661d)

dspy.GEPA has no max_steps parameter: the call raised TypeError on every run and silently fell back to MIPROv2, so the GEPA path advertised in the README could never execute. GEPA also requires exactly one budget parameter and a reflection LM, and its GEPAFeedbackMetric protocol calls the metric with (gold, pred, trace, pred_name, pred_trace).

Fix: pass max_full_evals=iterations and reflection_lm; extend skill_fitness_metric to the GEPA calling convention, returning Prediction(score, feedback) with deterministic missing-terms feedback for predictor-level reflection calls and a plain float everywhere else (holdout and MIPROv2 behavior unchanged).

Enhancement — generation budget for local endpoints (5b7fbd9)

Reasoning models served by local OpenAI-compatible endpoints often sit behind low server-side max_tokens defaults: the thinking phase consumes the whole budget and content comes back empty ('LM returned an empty or null response' on every rollout). New --max-tokens / --temperature options apply to both the eval LM and GEPA's reflection LM; defaults unchanged.

Tests

145 → 154 (9 new: GEPA-visibility of the skill text, full-file vs bare-body constraint regression, metric contract incl. the max_steps regression), all green.

🤖 Generated with Claude Code

enzo-adami and others added 5 commits August 4, 2026 10:19
…lve it

GEPA's candidate space is {name: pred.signature.instructions} over
named_predictors() (dspy/teleprompt/gepa/gepa.py:126,599). SkillModule kept
the skill body in a plain attribute passed as an InputField, which GEPA
never mutates — optimized_module.skill_text always returned the original
text, so the saved evolved_skill.md could never contain the evolution.

Move the skill body into the predictor's signature instructions via
with_instructions(), and make skill_text a read-through property over
named_predictors() so extraction reflects the evolved candidate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
validate_all() was called with skill['body'] / evolved_body, but
_check_skill_structure() looks for YAML frontmatter — which load_skill()
strips from body. The structure constraint therefore failed on every run
and no evolved candidate could ever pass the gate ('not deploying').

Validate skill['raw'] for the baseline and the reassembled evolved_full
for the candidate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y runs

dspy.GEPA has no `max_steps` parameter — the previous call raised
TypeError on every run and silently fell back to MIPROv2, so the GEPA
path advertised by the README could never execute. GEPA also requires
exactly one budget parameter and a reflection LM, and its
GEPAFeedbackMetric protocol calls the metric with
(gold, pred, trace, pred_name, pred_trace).

- Pass max_full_evals=iterations and reflection_lm (optimizer model).
- Extend skill_fitness_metric to the GEPA calling convention; return
  Prediction(score, feedback) with deterministic missing-terms feedback
  for predictor-level reflection calls, plain float everywhere else
  (holdout and MIPROv2 behavior unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reasoning models served by local OpenAI-compatible endpoints often sit
behind low server-side max_tokens defaults: the thinking phase consumes
the whole budget and content comes back empty ('LM returned an empty or
null response' on every rollout). Let callers set an explicit generation
budget and temperature for both the eval LM and GEPA's reflection LM.
Defaults unchanged (kwargs only passed when provided).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…al local endpoints

On a serial local endpoint (decode-concurrency 1), GEPA's parallel
rollouts queue behind each other; once queue depth times per-request
latency exceeds the client timeout, every in-flight call times out in
cascade and litellm's default retries re-pile the queue (observed:
'Average Metric: 0.00 / 0' with APITimeoutError on every rollout, while
the server kept generating for closed sockets). Let callers serialize
evaluation (GEPA num_threads) and bound per-request timeout/retries.
Defaults unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant