fix(evolution): report skipped holdout as N/A + preserve GEPA candidate (t_ac619234) - #165
Open
RomanXSad wants to merge 9 commits into
Open
Conversation
…7%, shorts +18.1%)
…t_316c92c4) Root cause (t_3832d98c): holdout eval (24x3x2=144 LLM calls, last phase) with no per-call timeout and no checkpointing blew the gtimeout-480 budget 3x. This commit: - make_lm(): dspy.LM now gets timeout=60 (litellm kwarg) + num_retries=2, so a stalled API call raises quickly instead of hanging 600s+ (litellm default) and getting SIGKILLed with no output. - evaluate_holdout(): refactored holdout loop with in-process wall-clock budget (--max-budget, default 480, passed the remaining time) -> returns a PARTIAL result (budget_exceeded=True) instead of being killed. - Persistent score cache output/<skill>/holdout_scores.json keyed by (eval model, skill-body hash, program, example, sample): warm runs skip the 144-call holdout phase entirely (checkpointing for the 480s budget). - metrics.json gains holdout_complete/examples/cache_hits/calls_made. - New regression tests (7) in tests/skills/test_timeout_regression.py. Note: this file also carries in-flight harness edits from prior fix tasks (dataset load_or_generate, seed, GEPA 5-arg metric) that were uncommitted.
…handling (t_47e91803) Constraints FAILED (esc-20260731-201857-1474): growth_limit hard-rejected a genuinely better candidate (+71.3% size, +0.035 valset) against the 50% soft cap. _check_growth now accepts a measured improvement and waives the soft cap up to a new hard cap (max_prompt_growth_hard=1.0) when the delta clears growth_waiver_min_improvement=0.03. evolve_skill.py measures valset improvement only when the growth gate trips (12-example cap, median aggregation) and re-validates with it. error 1: EMFILE (OSError 24) crashes at artifact writes and the holdout eval surfaced as bare exit 1, mislabeled 'LLM provider issue' by cron-evolve.sh. New evolution/core/errors.py defines the exit-code contract (1 generic, 2 EMFILE, 3 eval error); writes go through write_text_guarded() (exit 2 with ulimit hint on EMFILE) and the holdout phase classifies exceptions. Baseline validation now checks the full artifact (skill['raw']), fixing the body-vs-full asymmetry at evolve_skill.py:133. Tests: growth boundary (exact 50%), 0-char baseline, negative growth, waiver grant/deny/hard-cap, validate_all end-to-end; classifier tests for the EMFILE path; guard tests for write_text_guarded and holdout helpers.
…erruns (t_316c92c4) Verified failure: first verification run under the exact cron invocation exited 124 — cold-cache GEPA consumed 439s of the 480s budget (141 rollouts x 4.14s after the skill-text edit invalidated ~/.dspy_cache), leaving the holdout phase 71s and the process SIGKILLed mid-call. GEPA's iteration loop is not hookable, so: - start_budget_watchdog(): daemon exits max_budget - 10s with exit 124 (gtimeout-compatible -> cron-evolve.sh auto-retries) after writing a budget_checkpoint.json (phase, elapsed, timed_out). ~/.dspy_cache makes the retry warm/fast instead of a zero-output SIGKILL. - Holdout skip: if < 60s of budget remains, skip the holdout eval, keep the baseline skill text, and finish cleanly with holdout_skipped=true in metrics.json (no improvement claim) instead of dying at 480s. - Holdout headroom: holdout budget = remaining - 30s reserve so metrics and the report always have time to write after the phase. - Checkpoint updated at phase boundaries (gepa / validate / holdout). - Regression tests: exit-code 124 compatibility, watchdog/holdout layering, checkpoint persistence (10 tests total in test_timeout_regression.py).
…or, t_63240783) Growth gate compared the evolved FULL artifact (frontmatter+body) against the baseline BODY only, inflating the ratio (gmail-monitor +100.5% vs +69.8% body-vs-body, hard cap +100%) and rejecting a real +0.045 valset improvement by 0.5%. Add growth_text param to validate_all/_check_growth so the gate compares evolved body vs baseline body in both the initial check and the waiver re-validation. Failure path now also writes a timestamped dir (baseline_skill.md + evolved_FAILED.md + metrics.json) so cron-evolve.sh can diff the CURRENT run's artifacts instead of falling back to a stale identical pair (has_diff=0).
…s (t_63240783) - Waiver re-validation now also compares body-vs-body (growth_text=evolved_body); previously only the initial check did, so the waiver path still used the frontmatter-inflated ratio (+82.6% vs +51.8%). - Failure path writes a timestamped dir (baseline + evolved_FAILED + metrics) so cron diffs the current run's artifacts, not a stale identical pair. - Fresh dspy.context per holdout iteration (Python >=3.14 one-shot _GeneratorContextManager) — pre-existing working-tree fix, required for the holdout eval to run at all.
…xceeded (t_f8e3ec7f) The waiver-failure message always said 'improvement < waiver threshold' even when the waiver WAS applied (improvement >= threshold) and the real cause was the +100% hard-cap breach (gmail-monitor: +0.065 improvement, +115.6% growth was reported as '0.065 < 0.030'). Message-only fix — no threshold semantics changed. Regression test added.
…te (t_ac619234) The budget-guard holdout skip (remaining < 60s) used to hardcode 0.0 scores and overwrite GEPA's candidate with a byte-identical baseline copy, so a skipped holdout surfaced as a fabricated '+0.000 (+0.0%)' / 'no improvement' verdict (pipeline-verification 2026-08-02). - handle_holdout_skip(): preserves candidate_full before the baseline overwrite; returns placeholder scores + skipped holdout_result - build_metrics(): baseline_score/evolved_score/improvement are NULL when holdout_skipped (N/A downstream), never 0.0; adds holdout_candidate_preserved - results table + final summary render 'N/A' / 'holdout skipped' instead of 0.000 when the holdout was skipped - evolved_candidate.md is written alongside the baseline-copy evolved_skill.md so a later budget-raised re-run can diff/reuse GEPA's candidate - new tests: tests/skills/test_holdout_skip_reporting.py (3 tests, all pass)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the fabricated zero-improvement signal when the holdout eval is skipped by the budget guard (
remaining < HOLDOUT_SKIP_THRESHOLD_SECONDS). Verified 2026-08-02 on pipeline-verification (t_43f1f626 / t_a752e135): the skip path hardcoded0.0scores and overwrote GEPA's candidate with a byte-identical baseline copy (md5 0a959721both), so the run surfaced as+0.000 (+0.0%)/ "no improvement" — a placeholder, not a measurement (the metric is real: same skill text scored 0.423 on 2026-07-30).Changes (t_ac619234)
evolution/skills/evolve_skill.pyhandle_holdout_skip()(new helper) — preserves GEPA's candidate incandidate_fullbefore the baseline overwrite; returns placeholder scores +skipped: trueresult. The deployed artifact stays the baseline (byte-identical →has_diff=0), mirroring the 8b regression guard.build_metrics()(new helper) — whenholdout_skipped,baseline_score/evolved_score/improvementare written as null (N/A), never0.0; addsholdout_candidate_preserved.Holdout Score: N/A | N/A | —and the final summary says "Holdout skipped — no score measured" instead of0.000/ "did not improve".evolved_candidate.mdwith GEPA's real candidate alongside the baseline-copyevolved_skill.md, so a later budget-raised re-run can diff/reuse it.tests/skills/test_holdout_skip_reporting.py(new) — 3 tests pinning: candidate survives the skip, metrics are null when skipped (incl. JSON round-trip), non-skipped runs keep real float scores.Verification
pytest tests/skills/test_holdout_skip_reporting.py -v→ 3 passedpytest tests/skills/→ 193 passed; the 10 failures are pre-existing (untracked skill-artifact tests + a timing-flaky 0.15s-budget race test that fails identically without this change — A/B verified)bash -nclean on the companion cron-evolve.sh change (separate dotfiles repo, not in this PR)Companion change (local dotfiles repo
~/.hermes/scripts/cron-evolve.sh)Section 5a now reads
holdout_skippedfrommetrics.json, forces scores to N/A, gates the PR branch (HOLDOUT_SKIPPED != 1), and emits an explicit "holdout skipped (budget guard)" verdict + ticket note — a skipped holdout can never be misread as "no improvement". Dry-run harness:cron-holdout-dryrun.sh(all checks pass, incl. adversarial HAS_DIFF=1 + skipped → no PR).