Skill evolution is a no-op: optimized_module.skill_text is the unchanged baseline

[sunghunkwag]

The skill-evolution pipeline cannot, by construction, modify the skill text it claims to evolve. The "evolved" SKILL.md written to output/ is byte-identical to the input.

Root cause. In skill_module.py, the skill body is stored as a plain instance attribute and passed as a DSPy InputField value:

self.skill_text = skill_text                 # plain attribute
...
skill_instructions=self.skill_text,          # passed as an input VALUE

DSPy optimizers (GEPA, MIPROv2, BootstrapFewShot) only mutate a Predict submodule's signature.instructions and demos. A plain attribute is never registered as a parameter, so compile() never touches it. evolve_skill.py:183 then reads the untouched attribute back:

evolved_body = optimized_module.skill_text   # == baseline, always

Reproduction (no API key needed, dspy 3.x):

import dspy
class TaskWithSkill(dspy.Signature):
    skill_instructions: str = dspy.InputField()
    task_input: str = dspy.InputField()
    output: str = dspy.OutputField()
class SkillModule(dspy.Module):
    def __init__(self, skill_text):
        super().__init__(); self.skill_text = skill_text
        self.predictor = dspy.ChainOfThought(TaskWithSkill)
m = SkillModule("ORIGINAL BODY")
print([n for n,_ in m.named_parameters()])   # ['predictor.predict']
print("skill_text" in dict(m.named_parameters()))  # False
print(m.deepcopy().skill_text == m.skill_text)     # True  -> never changes

Consequence for the Phase-1 report. The reported 0.408 → 0.569 (+39.5%) cannot have come from a changed skill. With BootstrapFewShot (per the report) the only thing optimized is bootstrapped demos attached to the predictor — and reassemble_skill() discards them, writing only frontmatter + unchanged body. The headline gain is a transient demo-augmented surrogate that is never saved or deployed.

Proposed fix (verified to round-trip). Make the skill body the optimizable signature.instructions, and read it back from there:

class SkillModule(dspy.Module):
    def __init__(self, skill_body: str):
        super().__init__()
        sig = dspy.Signature("task_input -> output", instructions=skill_body)
        self.predictor = dspy.ChainOfThought(sig)

    def forward(self, task_input: str) -> dspy.Prediction:
        result = self.predictor(task_input=task_input)
        return dspy.Prediction(output=result.output)

    @property
    def skill_body(self) -> str:
        return self.predictor.predict.signature.instructions

# evolve_skill.py
evolved_body = optimized_module.skill_body   # was optimized_module.skill_text

Verified: after an instruction optimizer rewrites predictor.predict.signature.instructions, skill_body returns the new text — the optimized artifact is now the saved artifact. This fix is a prerequisite for the two companion issues (the bag-of-words fitness metric / dead LLMJudge, and the silently-swallowed dspy.GEPA(..., max_steps=...) call) to have any effect; I'll cross-link them here once filed.

[coden-gutes]

Verified independently + complete working patch set (dspy 3.2.1).

I reproduced the no-op exactly (2026-07-15): skill_text is NOT in named_parameters() (only predictor.predict), and deepcopy().skill_text == skill_text is always True. So every MIPROv2 trial scored identically to baseline — the run "completes" but evolves nothing.

The fix in #141 works. Three patches make the pipeline actually deploy an evolved skill:

1. evolution/skills/skill_module.py — make the body the optimizable instructions (the #141 fix):

class SkillModule(dspy.Module):
    def __init__(self, skill_body: str):
        super().__init__()
        sig = dspy.Signature("task_input -> output", instructions=skill_body)
        self.predictor = dspy.ChainOfThought(sig)
    def forward(self, task_input: str):
        return dspy.Prediction(output=self.predictor(task_input=task_input).output)
    @property
    def skill_text(self):
        return self.predictor.predict.signature.instructions

Verified: after an optimizer rewrites signature.instructions, skill_text returns the NEW text.

2. evolution/skills/evolve_skill.py — fix the validator false-positive (#74/#110/#119): the evolved check validated evolved_body (no frontmatter) so skill_structure always failed. Validate the reassembled full file instead:

baseline_constraints = validator.validate_all(skill["raw"], "skill")
...
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

3. evolution/skills/skill_module.py — reassemble_skill nested-frontmatter (#119): strip any leading frontmatter the optimizer may have written into the body before re-wrapping.

With all three, an evolved skill that beats baseline now passes constraints and is written as a deployable evolved.md (not _FAILED.md). Note: the synthetic eval set gives weak signal (#12) — real gains need --eval-source sessiondb or a golden set. Also optuna must be installed for MIPROv2, and dspy.GEPA(..., max_steps=...) needs max_full_evals= on dspy 3.x (GEPA also requires a 5-arg metric, so MIPROv2 is the working path).

[weijyhnjean]

Independent confirmation on a different stack: local vLLM serving Qwen/Qwen3.6-35B-A3B-FP8 (fully local, no cloud API), dspy 3.2.1, hermes-agent-self-evolution main @ 0a929e3.

Reproduced exactly as described — ran a full GEPA optimization (30 rollouts, genuine reflective mutation, real Pareto-front score improvements over 3 iterations, ~322s wall time) against a custom test skill, then diff'd the baseline against output/<skill>/evolved_FAILED.md: byte-identical. Confirms this isn't optimizer-specific or provider-specific — tried both MIPROv2 (via the GEPA fallback path) and, after patching around the two issues below, genuine GEPA. Same null result both times, exactly matching the named_parameters() repro in this issue.

Two more DSPy-3.2 API breaks I hit getting to the point where I could confirm the above, on top of the max_steps rename mentioned in #119:

1. dspy.GEPA's metric now requires the 5-arg feedback-metric signature (gold, pred, trace, pred_name, pred_trace) returning a ScoreWithFeedback (Prediction with .score/.feedback). The shipped skill_fitness_metric is still the old 3-arg plain-float signature, so dspy.GEPA(metric=skill_fitness_metric, ...) raises TypeError: GEPA metric must accept five arguments... before even getting to the max_steps issue for anyone who fixes that one first. Worked around locally with a thin wrapper that adapts the existing (already-implemented-but-unused) LLMJudge class into the new shape.
2. dspy.GEPA also now requires an explicit reflection_lm — it no longer silently reuses dspy.settings.lm. Constructor raises AssertionError: GEPA requires a reflection language model... without it.

Neither blocks the fix proposed here (skill_body as signature.instructions) — just flagging them since anyone patching max_steps next will hit both immediately afterward, before ever reaching the actual no-op bug this issue describes.

[TurtleMcTurtle]

Fixed in commit c5138ca ().\n\nWhat changed in : The skill text is now embedded as the string instead of being passed as an value. This means GEPA's instruction-mutation logic can actually reach and modify the skill content during optimization.\n\nBefore (broken):\n\n\nAfter (fixed):\n\n\nThe property now reads from the live Signature, so returns the GEPA-mutated version.\n\nAdditionally: fitness metric updated to 5-arg GEPA-compatible signature, GEPA constructor fixed for DSPy 3.3.0, and 96 new tests added (241 total passing).