Skip to content

evolve_skill always fails skill_structure constraint — validator checks bare body instead of reassembled skill #169

Description

@ycheng87

Summary

Running python -m evolution.skills.evolve_skill always ends with the skill_structure constraint failing — even when the evolved output is a complete, valid SKILL.md with intact YAML frontmatter:

✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
✗ Evolved skill FAILED constraints — not deploying

Root cause

In evolution/skills/evolve_skill.py (~lines 181–186):

evolved_body = optimized_module.skill_text
evolved_full = reassemble_skill(skill["frontmatter"], evolved_body)

# ── 7. Validate evolved skill ───────────────────────────────────────
evolved_constraints = validator.validate_all(evolved_body, "skill", ...)

The validator receives evolved_body — the bare body without the YAML frontmatter — while _check_skill_structure in evolution/core/constraints.py requires the text to start with --- and contain name: / description: in the first 500 chars. A bare body never contains frontmatter, so the check always fails regardless of how good the evolved skill actually is.

Repro

  1. python -m evolution.skills.evolve_skill --skill systematic-debugging --iterations 3 --eval-source synthetic
  2. The optimizer completes (best score reported), then validation always reports the missing-frontmatter failure.
  3. The saved output/systematic-debugging/evolved_FAILED.md is actually a complete file with valid frontmatter — in our run it is byte-identical to the baseline (0 diff), which also confirms the failure is in validation, not in generation.

Suggested fix

Validate the reassembled artifact instead of the bare body:

evolved_full = reassemble_skill(skill["frontmatter"], evolved_body)
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["body"])

(Note: baseline_text comparisons in validate_all may also need the reassembled baseline — reassemble_skill(skill["frontmatter"], skill["body"]) — so size/growth gates compare like with like.)

Environment

  • hermes-agent-self-evolution v0.1.0 (main)
  • dspy 3.3.0, optuna 4.9.0, Python 3.11.15
  • Model: deepseek-v4-flash via an OpenAI-compatible endpoint — the bug reproduces with any model since it lives in validation, not generation.

Root cause identified by 宇航员 (Xiaomo), the reporter's Hermes agent; filed by ycheng87.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions