feat(fitness): objective ground-truth verifier for skill evolution (reference: arxiv)

[MaxFreedomPollard]

The problem

Phase 1's gate is "at least one skill measurably improved." Right now that measurement can't be trusted, because the metric the optimizer maximizes (and the holdout is scored with) is keyword overlap against the rubric text:

# evolution/core/fitness.py
overlap = len(expected_words & output_words) / len(expected_words)
score = 0.3 + (0.7 * overlap)

This rewards echoing the rubric's vocabulary, not being right. LLMJudge exists but is dead code (imported in evolve_skill.py, never called), and even judge scoring is subjective: it never checks whether an answer is actually true. The Phase 1 validation report is honest about this and lists replacing the heuristic as its own next step.

So every reported improvement so far is unfalsifiable. Nothing downstream (benchmark gates, significance checks like #135/#136, Phase 2's tool-selection accuracy) means anything until the number being optimized measures reality.

What this adds

A third, strongest tier of fitness signal: objective verifiers that grade outputs against checkable ground truth, plus a working reference implementation for the arxiv skill (the same skill the Phase 1 validation report used).

Signal	How it scores	Trust
Objective verifier (new)	Compares output to verifiable facts; pure Python	Not gameable by rubric echoing, reproducible, $0 per grade
LLM-as-judge	Model grades a paraphrase on a rubric	Subjective, circular
Keyword overlap (current)	Lexical overlap with the rubric	Weakest

evolution/core/verifier.py

• Verifier ABC: a verifier owns both halves of the evaluation contract, build_dataset() (tasks whose answers are verifiable facts) and score() (grade an output against those facts). They must travel together, since the grader can only score tasks it knows the truth for.
• Registry (@register_verifier, get_verifier(skill_name)).
• verifier_metric() adapts a verifier to every metric calling convention in the pipeline: plain float for MIPROv2 and holdout comparison, and score-plus-feedback (dspy.Prediction(score=..., feedback=...)) when GEPA asks about a specific predictor. The verifier's feedback text ("expected ID 1706.03762, response said 1706.03799") is exactly the reflection signal GEPA mutates on.

evolution/verifiers/arxiv_verifier.py

Grades four kinds of verifiable facts across 17 landmark ML papers (word2vec through the GEPA paper itself): arXiv IDs, exact titles, first authors, and submission years. Grading is regex plus normalization, no LLM calls.

Score shaping is monotone and anti-gaming:

Output	Composite
Correct and concise	1.00
Correct but hedged across several candidate IDs	0.75
Confidently wrong (an ID was given, it's the wrong one)	0.50
Evasive or empty	0.00

Details that matter:

• Hedging gets half credit. Listing several candidate IDs where one happens to be right is not a usable answer, and full credit would reward shotgunning.
• Conciseness only counts when an answer is present. A brief evasion can never outscore a verbose correct answer (a test locks this in).
• Year tasks are solvable from the ID alone (the YYMM prefix), so they measure whether the skill teaches the arXiv ID format. That's a property a skill edit can genuinely flip, which is exactly the kind of headroom evolution needs.
• ID-shaped substrings are stripped before year matching, so quoting 2005.14165 doesn't get misread as the year 2005.

The embedded facts are historical constants, and every one is re-checkable against the live arXiv API:

$ python -m evolution.verifiers.arxiv_verifier --validate
All 17 papers verified against the live arXiv API.

All 51 facts (17 papers x title/author/year) passed at commit time.

Wiring (evolve_skill.py)

New --fitness auto|verifier|keyword flag, default auto: use the objective verifier when one is registered for the skill, otherwise fall back to current behavior. When a verifier is active it also supplies the eval dataset (deterministic sample, 50/25/25 split, saved to datasets/skills/<skill>-verifier/).

# Objective fitness picked automatically
python -m evolution.skills.evolve_skill --skill arxiv --iterations 10

# See how the grader treats correct, wrong, and evasive answers
python -m evolution.verifiers.arxiv_verifier --demo

Deliberate scope boundaries

• Does not touch GEPA instantiation, evolved-text extraction, or the frontmatter validator; fix(phase1): GEPA-compatible SkillModule + LLM-as-judge metric + constraint fixes #137, fix(skills): extract evolved instruction with overfit/collapse guard #146, fix: GEPA 3.2.x compatibility + symlink resolver + validator false-positive #142, Fix evolve_skill validation and assembly bugs (fixes #119) #140 and others own those fixes, and this composes with whichever of them land. The metric adapter already speaks GEPA's feedback protocol, so it plugs straight into a fixed optimizer loop.
• Does not replace judge or keyword scoring; the --fitness selector leaves room for an LLM-as-judge mode (as in fix(phase1): GEPA-compatible SkillModule + LLM-as-judge metric + constraint fixes #137) for skills where no objective verifier is possible.
• Adding a verifier for another skill is subclass-and-register; tool-selection accuracy (Phase 2) and test suites (Phase 4) are both objectively checkable, so this interface is the substrate those phases need anyway.

Test plan

• 40 new tests (tests/verifiers/test_arxiv_verifier.py, tests/core/test_verifier_registry.py): grading correctness for all four fact kinds, the anti-gaming behaviors, dataset determinism and gradability, metric adapter under all three calling conventions, edge cases (empty output, unknown task, versioned IDs).
• Full suite: 183 passed (143 existing + 40 new), offline; the live API is only touched by the --validate flag.
• Manual: --validate (17/17 papers verified), --demo, and evolve_skill --dry-run in both auto-verifier and forced-keyword modes.

No new dependencies (stdlib urllib + xml.etree for the validation fetch; click/rich already in use).