Skip to content

bug: skill_structure validation always fails on evolved skills (validates bare body, not reassembled frontmatter+body) #171

Description

@lucky20260806

Bug

In evolution/skills/evolve_skill.py step 7, the evolved skill is validated with the bare body (no YAML frontmatter):

evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])

But ConstraintValidator._check_skill_structure requires the text to start with --- and contain name: / description: — which only exist in the reassembled full skill (evolved_full), not the bare body. Result: every evolved skill fails skill_structure, even when the reassembled file (frontmatter + body) is perfectly valid.

Repro

  1. python -m evolution.skills.evolve_skill --skill lucky-agents --iterations 3 --eval-source synthetic
  2. Observe: ✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
  3. The saved output/<skill>/evolved_FAILED.md actually HAS valid frontmatter — because it was reassembled before saving, but validated before reassembly.

Fix

Validate the reassembled full skill instead (same text that gets saved/deployed):

evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

Also the baseline check in step 3 has the same latent issue (validate_all(skill["body"], "skill")), though it only warns.

Verified: with this fix, a 10-iteration run passes all 4 constraints (size_limit, growth_limit, non_empty, skill_structure).

Environment

  • Windows 10, Python 3.11, dspy 3.3.0, gepa 0.1.1
  • Tested with DeepSeek as the LLM provider (openai/deepseek-chat) — the bug is provider-independent since it's a pure string-shape issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions