Summary
Running python -m evolution.skills.evolve_skill always ends with the skill_structure constraint failing — even when the evolved output is a complete, valid SKILL.md with intact YAML frontmatter:
✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
✗ Evolved skill FAILED constraints — not deploying
Root cause
In evolution/skills/evolve_skill.py (~lines 181–186):
evolved_body = optimized_module.skill_text
evolved_full = reassemble_skill(skill["frontmatter"], evolved_body)
# ── 7. Validate evolved skill ───────────────────────────────────────
evolved_constraints = validator.validate_all(evolved_body, "skill", ...)
The validator receives evolved_body — the bare body without the YAML frontmatter — while _check_skill_structure in evolution/core/constraints.py requires the text to start with --- and contain name: / description: in the first 500 chars. A bare body never contains frontmatter, so the check always fails regardless of how good the evolved skill actually is.
Repro
python -m evolution.skills.evolve_skill --skill systematic-debugging --iterations 3 --eval-source synthetic
- The optimizer completes (best score reported), then validation always reports the missing-frontmatter failure.
- The saved
output/systematic-debugging/evolved_FAILED.md is actually a complete file with valid frontmatter — in our run it is byte-identical to the baseline (0 diff), which also confirms the failure is in validation, not in generation.
Suggested fix
Validate the reassembled artifact instead of the bare body:
evolved_full = reassemble_skill(skill["frontmatter"], evolved_body)
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["body"])
(Note: baseline_text comparisons in validate_all may also need the reassembled baseline — reassemble_skill(skill["frontmatter"], skill["body"]) — so size/growth gates compare like with like.)
Environment
- hermes-agent-self-evolution v0.1.0 (main)
- dspy 3.3.0, optuna 4.9.0, Python 3.11.15
- Model: deepseek-v4-flash via an OpenAI-compatible endpoint — the bug reproduces with any model since it lives in validation, not generation.
Root cause identified by 宇航员 (Xiaomo), the reporter's Hermes agent; filed by ycheng87.
Summary
Running
python -m evolution.skills.evolve_skillalways ends with theskill_structureconstraint failing — even when the evolved output is a complete, valid SKILL.md with intact YAML frontmatter:Root cause
In
evolution/skills/evolve_skill.py(~lines 181–186):The validator receives
evolved_body— the bare body without the YAML frontmatter — while_check_skill_structureinevolution/core/constraints.pyrequires the text to start with---and containname:/description:in the first 500 chars. A bare body never contains frontmatter, so the check always fails regardless of how good the evolved skill actually is.Repro
python -m evolution.skills.evolve_skill --skill systematic-debugging --iterations 3 --eval-source syntheticoutput/systematic-debugging/evolved_FAILED.mdis actually a complete file with valid frontmatter — in our run it is byte-identical to the baseline (0 diff), which also confirms the failure is in validation, not in generation.Suggested fix
Validate the reassembled artifact instead of the bare body:
(Note:
baseline_textcomparisons invalidate_allmay also need the reassembled baseline —reassemble_skill(skill["frontmatter"], skill["body"])— so size/growth gates compare like with like.)Environment
Root cause identified by 宇航员 (Xiaomo), the reporter's Hermes agent; filed by ycheng87.