skillcraft: add reflective optimization benchmark

skillcraft/results/.gitignore

@@ -5,3 +5,5 @@
 !REPORT.zh_CN.md
 !multi_family_compare/
 !multi_family_compare/**
+!gepa_reflective_optimization/
+!gepa_reflective_optimization/**

skillcraft/results/README.md

@@ -1,103 +1,80 @@
 # Evaluation Results — SkillCraft
 
-This directory stores the SkillCraft benchmark artefacts for
-`trpc-agent-go`'s `evolution` loop.
+This directory contains two complementary evaluation tracks:
 
-The old headline in this folder came from a single historical run. The
-current source of truth is the latest **three-run full compare batch**:
+- the online asynchronous [`evolution` report](REPORT.md), with a
+  [Chinese version](REPORT.zh_CN.md); and
+- the pure-Go reflective optimizer's
+  [five-family report](gepa_reflective_optimization/REPORT.md), with a
+  [Chinese version](gepa_reflective_optimization/REPORT.zh_CN.md).
 
-- [`full_compare_run1`](full_compare_run1)
-- [`full_compare_run2`](full_compare_run2)
-- [`full_compare_run3`](full_compare_run3)
-- Aggregate JSON:
-  [`tools/full_compare_analysis.json`](tools/full_compare_analysis.json)
-- Aggregate script:
-  [`tools/aggregate_runs.py`](tools/aggregate_runs.py)
+They answer different questions. Online evolution learns between tasks. The
+optimizer searches over a frozen skill offline, validates candidates, and uses
+an untouched holdout before any candidate can be promoted.
 
-## Headline Numbers
+## Current Reflective-Optimization Result
 
-Configuration:
+The latest experiment uses the same five SkillCraft families and six scales in
+three paired arms: `baseline`, `evolution`, and `optimized_evolution`. Three
+fresh root seeds produce 90 tasks per arm and 270 arm-cases in total. Search,
+review, and task execution all requested the self-deployed GLM-5.2 route
+(`glm52`).
 
-- Agent / reviewer model: `gpt-4o-mini`
-- Task set: `openmeteo-weather`, `recipe-cookbook-builder`,
-  `world-bank-economic-snapshot`
-- Difficulty scales: `e1,e2,e3,m1,m2,h1` per family
-- Total tasks per run: `18`
-- Compare runs: `3`
-- Max tool iterations: `24`
-- Warm-start seed:
-  [`tools/clean_skill_seed`](tools/clean_skill_seed)
-- Prompt overview cap: `8`
-
-Aggregate over the three runs:
-
-| Metric | Baseline | Evolution | Δ (Evolution − Baseline) |
+| Metric | Baseline | Evolution | Optimized evolution |
 | --- | ---: | ---: | ---: |
-| Mean pass rate | 90.74% | 90.74% | 0.00pp |
-| Pass-rate stddev | 8.49pp | 3.20pp | - |
-| Mean end-to-end tokens/task | 169,888.61 | 145,980.13 | -23,908.48 |
-| End-to-end token stddev | 81,007.55 | 24,363.25 | 57,153.77 |
-| Mean agent tokens/task | 169,888.61 | 131,990.93 | -37,897.68 |
-| Mean duration delta | - | - | +21.14s |
-
-Key takeaways:
-
-- Evolution no longer has a clear multi-run pass-rate advantage on this
-  benchmark; the three-run mean pass rate is identical to baseline.
-- Evolution is more stable in pass-rate variance than baseline.
-- Mean end-to-end tokens improved only because one baseline run hit
-  catastrophic weather loops. In the other two runs, evolution still
-  spent more end-to-end tokens than baseline.
-- Evolution exposed skills in every task (`SkillsOffered = 100%`) but
-  never invoked `skill_load` in any of the three runs.
-
-## Focused Diagnostics
+| Pass rate | 97.78% | 97.78% | **98.89%** |
+| Official quality | 95.98% | 95.96% | **97.21%** |
+| End-to-end tokens / task | **305,240** | 352,971 | 362,368 |
 
-The current plan specifically tracks `openmeteo-weather/e1,e2,m1`.
-Across the three runs:
+The global optimized arm passed the fixed aggregate gate, but that verdict is
+not sufficient to promote every overlay: most of its quality delta came from
+Pokémon completion variance even though Pokémon received no optimized skill.
+Pooling the two changed families gives a causal `6.77%` end-to-end token
+reduction. The individual candidate result is clearer still: Recipe passed all
+18 tasks in both arms, improved quality by 0.32 percentage points, and reduced
+end-to-end tokens by 14.75%, with a token reduction in every root seed. World
+Bank also kept pass and quality unchanged, but cost 3.29% more and was rejected.
 
-| Task | Baseline Passes | Evolution Passes | Baseline Mean E2E | Evolution Mean E2E | Main Observation |
-| --- | --- | --- | ---: | ---: | --- |
-| `openmeteo-weather/e1` | `T,T,T` | `T,T,T` | 489,459.00 | 80,643.67 | one baseline run blew up to 1.32M tokens; evolution stayed much lower |
-| `openmeteo-weather/e2` | `T,F,T` | `T,T,T` | 514,047.67 | 189,678.00 | evolution rescued one baseline failure, but still never called `skill_load` |
-| `openmeteo-weather/m1` | `T,T,T` | `T,T,T` | 107,458.33 | 215,112.33 | evolution stayed correct but was consistently more expensive |
+An earlier GPT-5.2 runtime matrix is retained as a negative cross-model
+portability test. It is not pooled with the same-model result. The offline
+process also caught and rejected a separate Recipe candidate that saved tokens
+on validation but regressed badly on untouched holdout.
 
-That changes the old diagnosis:
+The exact per-run, per-family, paired, and gate results are in
+[`gepa_reflective_optimization/glm_full_matrix_evidence.json`](gepa_reflective_optimization/glm_full_matrix_evidence.json).
 
-- the weather loop is no longer a stable deterministic evolution-only bug;
-- the current reproducible fact is instead: **explicit skill usage is
-  still absent**, and `world-bank-economic-snapshot/e2` is now the only
-  evolution failure that reproduces in all three runs.
-
-## Where To Look Next
-
-The latest logs suggest two active issues:
-
-1. `skill_load` remains unused even when relevant skills are offered.
-2. `worldbank_*` MCP tools occasionally hit 60-second request timeouts,
-   and those timeouts now dominate the recurring failure cluster.
-
-## Directory Layout
+## Evidence Layout
 
 ```text
 results/
 |-- README.md
-|-- REPORT.md
+|-- REPORT.md                         # online evolution report
 |-- REPORT.zh_CN.md
+|-- gepa_reflective_optimization/
+|   |-- README.md
+|   |-- REPORT.md                     # current optimizer report
+|   |-- REPORT.zh_CN.md
+|   |-- evidence.json                 # accepted Recipe repair
+|   |-- recipe_candidate.json
+|   |-- model_routing_evidence.json   # proves gpt-5.2 != glm52 routing
+|   |-- glm_full_matrix_evidence.json # same-model GLM-5.2 aggregate
+|   |-- full_matrix_evidence.json     # earlier GPT-5.2 aggregate
+|   +-- full_matrix/
+|       |-- generic_candidate_frozen_evidence.json
+|       |-- worldbank_candidate_frozen_evidence_v1.json
+|       +-- worldbank_candidate_frozen_evidence_v2.json
+|-- reflective_full_matrix_601/       # canonical result, raw runs ignored
+|-- reflective_full_matrix_602/
+|-- reflective_full_matrix_603/
+|-- reflective_glm_full_matrix_701/
+|-- reflective_glm_full_matrix_702/
+|-- reflective_glm_full_matrix_703/
 |-- tools/
-|   |-- extract_metrics.py
-|   |-- aggregate_runs.py
-|   |-- full_compare_analysis.json
-|   |-- metrics.json
-|   +-- clean_skill_seed/
-|-- full_compare_run1/
+|-- full_compare_run1/                # older online-evolution batches
 |-- full_compare_run2/
-|-- full_compare_run3/
-+-- ... older historical runs ...
++-- full_compare_run3/
 ```
 
-## Historical Note
-
-Older artefacts such as [`multi_family_compare`](multi_family_compare)
-are still useful for archaeology, but they should no longer be used as
-the top-level claim for whether evolution currently helps on SkillCraft.
+Older `multi_family_compare_*` and `full_compare_*` directories remain useful
+for regression history. They use different models, budgets, task sets, or
+protocols and must not be pooled with the current three-arm matrix.

skillcraft/results/gepa_reflective_optimization/README.md

@@ -0,0 +1,51 @@
+# Reflective Optimization Evidence
+
+This directory is the sanitized evidence package for the pure-Go reflective
+optimizer and its five-family SkillCraft replay.
+
+Read [`REPORT.md`](REPORT.md) or
+[`REPORT.zh_CN.md`](REPORT.zh_CN.md) for the experiment design, bad-case
+analysis, and scoped conclusion. The machine-readable evidence is split by the
+decision it supports:
+
+- [`evidence.json`](evidence.json) records the accepted repair of an existing
+  reviewer-produced Recipe skill. Across eight frozen holdout pairs it improved
+  quality from 95.50% to 98.35% and reduced agent tokens by 6.57%, with no pass
+  loss.
+- [`full_matrix/generic_candidate_frozen_evidence.json`](full_matrix/generic_candidate_frozen_evidence.json)
+  records a Recipe efficiency candidate that looked better on validation but
+  failed untouched holdout. It was rejected.
+- [`full_matrix/worldbank_candidate_frozen_evidence_v2.json`](full_matrix/worldbank_candidate_frozen_evidence_v2.json)
+  records a World Bank candidate that passed frozen confirmation but was later
+  rejected by the same-model operational replay.
+- [`glm_full_matrix_evidence.json`](glm_full_matrix_evidence.json) aggregates
+  the final same-model GLM-5.2 replay: three root seeds, five task families, six
+  scales, and three arms, for 270 arm-cases.
+- [`full_matrix_evidence.json`](full_matrix_evidence.json) retains the earlier
+  GPT-5.2 runtime replay as a negative cross-model portability result.
+- [`model_routing_evidence.json`](model_routing_evidence.json) records the check
+  showing that `gpt-5.2` and `glm52` are distinct routes on the configured
+  OpenAI-compatible endpoint.
+- [`recipe_candidate.json`](recipe_candidate.json) is the accepted Recipe
+  candidate used by both operational replays.
+
+Both operational aggregates include per-root family summaries and sanitized
+failed-case records so the report's attribution and bad-case analysis can be
+audited without publishing raw task workspaces or model transcripts.
+
+Search, frozen confirmation, and the final same-model replay requested the
+self-deployed GLM-5.2 route (`glm52`). In that replay, Recipe preserved a 100%
+pass rate, improved quality by 0.32 percentage points, and reduced end-to-end
+tokens by 14.75% across three consistently favorable root seeds. World Bank
+preserved pass and quality but increased tokens by 3.29%, so it was not
+promoted.
+
+The global aggregate marks the preregistered mechanical gate as eligible, but
+candidate attribution remains essential: most of the aggregate quality delta
+came from no-overlay Pokémon completion variance. Across the two changed
+families, the overlay reduced end-to-end tokens by 6.77%, but Recipe supplied
+all of the saving while World Bank added cost. The report therefore promotes
+Recipe and rejects World Bank as individual skill revisions.
+
+Raw model transcripts, task workspaces, endpoint configuration, credentials,
+and machine-specific paths are intentionally excluded.