4 bugs in evolve_skill pipeline — optimization works but evolved file is never deployed

[MrMooreUK]

Bug Report — hermes-agent-self-evolution pipeline

Repo: https://github.com/NousResearch/hermes-agent-self-evolution (or equivalent — adjust to actual location)
Date filed: 2026-06-15
Filed by: @MrMooreUK (local clone at ~/hermes-agent-self-evolution/, commit unknown — local working tree has uncommitted patches)
DSPy version tested: 3.2.1
Python version tested: 3.11.15 (uv-managed venv)
Reproduction env: macOS/Linux, OpenRouter as eval_model and optimizer_model

---

TL;DR

The evolve_skill end-to-end pipeline runs and produces an output, but has 4 bugs that prevent the evolved skill from ever being deployed, regardless of whether the optimization succeeded. Any user running this today will hit all 4 in sequence and conclude the tool is broken, when in fact 3 are local code bugs and 1 is a missing install-time dependency.

End-to-end optimization does work — 11 MIPROv2 trials, +6.72 score lift on the obsidian skill, ~2.5 min wall time. The evolved file is structurally valid but the validator and reassembler both have bugs that prevent deployment.

---

Bug 1 — validator.validate_all is called on body without its frontmatter

File: evolution/skills/evolve_skill.py
Line: 191
Severity: High (blocks all deployments)

Current code

evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])

Problem

evolved_body is the body of the evolved skill (no YAML frontmatter). The skill_structure constraint in evolution/core/constraints.py:150-174 checks:

has_frontmatter = text.strip().startswith("---")

Since evolved_body never starts with ---, has_frontmatter is always False, so the constraint always fails with "Skill missing: YAML frontmatter (---), name field, description field".

This failure is incorrect — the evolved body has no frontmatter by design, because the baseline's frontmatter is preserved separately and re-wrapped in reassemble_skill (line 187). The validator should be checking the reassembled full text (evolved_full), not the body alone.

Reproduction

Run any skill through evolve_skill. The evolved skill will always be saved with _FAILED.md suffix and never deployed, even if the optimization found a high-scoring candidate. Verify by:

ls output/<skill>/evolved_FAILED.md   # always present

Expected

The validator should be called on the full reassembled text (frontmatter + body), or skill_structure should be excluded from the body-only check.

Fix (suggested)

# Line 191: validate the reassembled full file, not the body
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

---

Bug 2 — reassemble_skill produces nested frontmatter blocks

File: evolution/skills/skill_module.py
Lines: 117-123
Severity: High (deployed output is malformed)

Current code

def reassemble_skill(frontmatter: str, evolved_body: str) -> str:
    """Reassemble a skill file from frontmatter and evolved body.
    Preserves the original YAML frontmatter (name, description, metadata)
    and replaces only the body with the evolved version.
    """
    return f"---\n{frontmatter}\n---\n\n{evolved_body}\n"

Problem

When the evolved body already contains its own YAML frontmatter (which it sometimes does — the optimizer occasionally writes a full skill file rather than just the body), the reassembler prepends the baseline frontmatter, producing nested ---\n...\n---\n\n---\n...\n---\n\n blocks. Markdown frontmatter parsers (including Hermes' loader) will read the first one and ignore the rest, but the file is technically malformed and any downstream tool that doesn't short-circuit on the first --- will misparse it.

Reproduction is the obsidian run — output/obsidian/evolved_FAILED.md has 2 nested frontmatter blocks because the optimizer wrote frontmatter into the body, then reassemble_skill added the baseline's frontmatter back on top.

Expected

Either (a) strip any leading frontmatter from evolved_body before reassembling, or (b) only prepend frontmatter if the body doesn't already start with ---\n.

Fix (suggested)

def reassemble_skill(frontmatter: str, evolved_body: str) -> str:
    # If the evolved body already has a frontmatter block, use it as-is and
    # don't prepend the baseline's.
    body = evolved_body
    if body.strip().startswith("---"):
        parts = body.split("---", 2)
        if len(parts) >= 3:
            body = parts[2].lstrip("\n")
    return f"---\n{frontmatter}\n---\n\n{body}\n"

---

Bug 3 — optuna is a runtime requirement but not a declared dependency

File: pyproject.toml (or wherever deps are declared — not yet found in this repo)
Severity: Medium (blocks first run after uv pip install -e .)

Problem

MIPROv2 (dspy.teleprompt.mipro_optimizer_v2) imports optuna lazily on first use. When MIPROv2 is selected as the optimizer fallback (because GEPA's API has changed in DSPy 3.x — see Bug 4), the first call to optimizer.compile() fails with:

ImportError: MIPROv2 requires optional dependency 'optuna'. Install it with `pip install dspy[optuna]`.

This means the pipeline cannot run end-to-end with the default pip install install. The error doesn't surface until MIPROv2 is invoked, so the dry-run pass (which doesn't invoke the optimizer) gives false confidence.

Expected

Either:

• Add optuna>=3.0 as a hard dependency in pyproject.toml, or
• Install the dspy[optuna] extras in setup.py / dependency declaration

Fix (suggested)

Add to [project].dependencies in pyproject.toml:

"optuna>=3.0",

OR change the import in DSPy to fail at import time rather than at first call (but that's a DSPy change, not in this repo's control).

---

Bug 4 — DSPy GEPA API mismatch causes silent fallback to MIPROv2 with no warning

File: evolution/skills/evolve_skill.py (GEPA call site, ~line 165-175)
Severity: Low (silent degradation, not a crash)

Problem

DSPy 3.x changed the GEPA API: GEPA(...) no longer accepts max_steps as a constructor argument. The current code passes max_steps=iterations and gets:

TypeError: GEPA.__init__() got an unexpected keyword argument 'max_steps'

The evolve_skill runner catches this (or it bubbles into the optimizer selection logic) and silently falls back to MIPROv2. No warning is logged to the user. A user who explicitly selected GEPA for its reflective/pareto-search behavior will get MIPROv2's bootstrap-and-propose behavior instead and not know it.

Reproduction

Select GEPA as the optimizer (the default). The run will execute with MIPROv2, and the user will see no indication that GEPA failed to load.

Expected

• Log a console.print("[yellow]WARNING: GEPA unavailable in DSPy X.Y — falling back to MIPROv2[/yellow]") before falling back
• OR fail loudly with a clear "GEPA API mismatch" error and a link to the DSPy changelog

Fix (suggested)

In evolve_skill.py where the optimizer is constructed, wrap the GEPA call in a try/except and emit a warning before falling back:

try:
    optimizer = GEPA(...)
except TypeError as e:
    console.print(f"[yellow]WARNING: GEPA unavailable ({e}); falling back to MIPROv2[/yellow]")
    optimizer = MIPROv2(...)

---

Bonus finding (not a bug, but worth knowing)

DSPy 3.2.1's dspy.LM defaults to max_tokens=1000, not 8192. The "8192 tokens, but can only afford 7094" error in our run was OpenRouter-side, not DSPy. The 4096 patch we added locally is still defensively correct for users on OpenRouter's free tier (where per-request cap is currently ~7000 for some models), but the framing in this repo's docs/README should not say "DSPy default is 8192" — that's a misconception.

---

Working reproduction (paste-ready)

# Prereqs: clone repo, cd in, uv venv .venv, uv pip install -e .
# Add OPENROUTER_API_KEY to .env.local

cd ~/hermes-agent-self-evolution
set -a && . ./.env.local && set +a
.venv/bin/python -m evolution.skills.evolve_skill \
  --skill obsidian \
  --iterations 5 \
  --eval-source golden \
  --dataset-path datasets/skills/obsidian/golden.jsonl \
  --hermes-repo /home/aaron/.hermes/hermes-agent \
  --eval-model openrouter/anthropic/claude-3.5-haiku \
  --optimizer-model openrouter/openai/gpt-4.1-mini

# Expected: 11 trials, best score ~40, but evolved_FAILED.md appears anyway (Bug 1)
# Expected: nested frontmatter in evolved_FAILED.md (Bug 2)
# Expected: ImportError if optuna not installed (Bug 3)
# Expected: silent MIPROv2 fallback, no warning (Bug 4)

---

Local workarounds applied (not part of upstream fix)

These are the patches I made to my local clone to get the run to execute at all. They are not the upstream fix — they're bandaids. The real fix is the 4 bugs above.

1. evolution/skills/evolve_skill.py:141 — added max_tokens=4096 to dspy.LM(...) to stay under OpenRouter free-tier per-request cap
2. evolution/core/dataset_builder.py:126 — same max_tokens=4096 patch
3. evolution/core/fitness.py:75 — same max_tokens=4096 patch
4. uv pip install optuna — to satisfy Bug 3

The local skill content at output/obsidian/evolved_FAILED.md is the post-bug-fix evolved file (frontmatter, +6.72 score, body nearly identical to baseline). It is NOT a deployable artifact.

---

Suggested upstream PR ordering

1. Bug 1 + Bug 2 (one PR — they're both about how evolved output is validated and assembled)
2. Bug 3 (one-line pyproject.toml change)
3. Bug 4 (warning message, ~5 lines)

Estimated total upstream work: ~30 min including tests. Will not affect optimizer behavior, only deployment correctness and observability.

---

Contact

For questions on this report, find me on the same channels where this skill was filed. Local repro env: ~/hermes-agent-self-evolution/ on Linux 6.8.0-124-generic, Python 3.11.15, uv-managed venv, DSPy 3.2.1, optuna 4.9.0 (installed post-ImportError).

[aranya-chatterjee]

I've verified all 4 bugs against current main. Planning a PR for Bug 1 + Bug 2 + Bug 3 (one PR) — tests written and passing locally. Note: Bug 4's warning already exists in current code (console.print(f"[yellow]GEPA not available ({e}), falling back to MIPROv2[/yellow]")), so no change needed there.