Impr

maxim-saplin · maxim-saplin · commit 0517f06dc9f9 · 2026-05-26T18:16:07.000+03:00
diff --git a/data/cross-ref/CONSOLIDATED_REPORT.md b/data/cross-ref/CONSOLIDATED_REPORT.md
@@ -25,7 +25,8 @@ Interpretation:
 - `ambiguous`, `unmatched`, and `excluded` rows stay visible in coverage and mapping review, but do not enter correlations.
 - Multiple external rows for one LLM Chess player are deduped by keeping the highest external score.
 - Pearson `r` measures linear fit; Spearman `rho` measures rank-order fit.
-- Release-controlled Elo removes the linear effect of release month from both sides before correlating residuals.
+- Raw Elo correlates external score directly with LLM Chess Elo.
+- Release-controlled Elo first predicts each side from release month, subtracts predicted from actual on both sides, then correlates the two residual lists.
 - OLS CV uses repeated 5-fold cross-validation over 3 seeds. Feature selection happens inside each training fold, not on the full target sample.
 
 ## Coverage Debt
@@ -46,8 +47,11 @@ The audit currently reports `reproducibility_status = pass`, `coverage_status =
 
 ## Evidence Pointers
 
-- Aggregate facts: [results/cross_ref_summary.json](results/cross_ref_summary.json)
-- Generated fact table: [results/cross_ref_report.md](results/cross_ref_report.md)
-- Mapping debt: [results/mapping_review.csv](results/mapping_review.csv)
+- Methodology and workspace guide: [README.md](README.md)
 - ECI summary: [results/eci_summary.json](results/eci_summary.json)
-- ARC summary: [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json)
+- ARC summary: [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json)
+- ECI report: [results/eci.html](results/eci.html)
+- ARC report: [results/arc_agi_2.html](results/arc_agi_2.html)
+- ECI coverage: [results/eci_coverage.csv](results/eci_coverage.csv)
+- ARC coverage: [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv)
+- Mapping sources: [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv)
diff --git a/data/cross-ref/README.md b/data/cross-ref/README.md
@@ -1,69 +1,116 @@
 # Cross-Ref Research Workspace
 
-## Essence
+## Purpose
 
-This workspace tests whether external benchmark scores line up with LLM Chess results after conservative model mapping.
+This workspace tests whether frozen external benchmark scores line up with LLM Chess outcomes after conservative model identity mapping.
 
-Current answer: Epoch ECI has a real positive relationship with LLM Chess Elo. ARC-AGI-2 is positive but weaker, noisier, and more blocked by model-identity ambiguity.
+This README is the operating guide. It should stay stable across result refreshes: use it for layout, artifact roles, commands, and methodology. Use [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) and the per-eval result artifacts for current findings.
+
+## Artifact Map
+
+| Artifact | Role |
+| --- | --- |
+| [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) | Cross-eval findings, current caveats, and research next steps. |
+| [results/eci_summary.json](results/eci_summary.json), [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) | Machine-readable per-eval facts used for reporting. |
+| [results/eci.html](results/eci.html), [results/arc_agi_2.html](results/arc_agi_2.html) | Human-readable per-eval reports. |
+| [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv) | Row-level coverage and inclusion surfaces. |
+| [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv) | Runtime model-identity source of truth. |
+| [mapping-research](mapping-research) | Evidence notes behind mapping decisions. |
+| [evals/eci/SOURCE.md](evals/eci/SOURCE.md), [evals/arc-agi-2/SOURCE.md](evals/arc-agi-2/SOURCE.md) | External source provenance and score semantics. |
+| [evals/eci/epoch_eci_apr_2026.csv](evals/eci/epoch_eci_apr_2026.csv), [evals/arc-agi-2/arc-agi-2-apr-2026.csv](evals/arc-agi-2/arc-agi-2-apr-2026.csv) | Frozen external source snapshots. |
 
 ## Methodology
 
-| Step | Rule | Why it matters |
-| --- | --- | --- |
-| Source parse | Keep external snapshots in `evals/`; parse the target score to a numeric column. | The benchmark score must be traceable to a frozen source file. |
-| Model mapping | Only `accepted`, `alias`, and `variant-compatible` rows enter analysis. `ambiguous`, `unmatched`, and `excluded` rows stay visible but out of correlations. | Weak identity guesses would move the numbers more than the math does. |
-| Join to chess | Mapped rows join to LLM Chess rows by canonical player name. | The row must exist in the current chess dataset. |
-| Deduplication | If several eval rows map to one chess player, keep the highest external score for that player. | One chess player should not get multiple votes in a correlation. |
-| Metric sample | Non-Elo chess metrics use `metric_analysis_rows_max_dedupe`. | These rows have chess metrics even when Elo is missing. |
-| Elo sample | Elo relationships use `elo_analysis_rows_max_dedupe`. | These rows have both an external score and non-null LLM Chess Elo. |
+### Pipeline
+
+1. Parse each frozen external snapshot into normalized rows with one numeric target score.
+2. Map external model names to canonical LLM Chess player names through the mapping CSVs.
+3. Include only `accepted`, `alias`, and `variant-compatible` mappings in statistics.
+4. Keep `ambiguous`, `unmatched`, and `excluded` rows visible in coverage outputs, but out of correlations.
+5. Join included mappings to LLM Chess rows by canonical player name.
+6. If multiple external rows map to one chess player, keep the highest external score for that player.
+7. Compute non-Elo metric relationships on `metric_analysis_rows_max_dedupe`.
+8. Compute Elo relationships on `elo_analysis_rows_max_dedupe`, which requires non-null external score and LLM Chess Elo.
+
+The mapping step is intentionally conservative. A plausible name match is not enough when reasoning mode, product tier, context window, benchmark system, or preview status could change the identity of the model being evaluated.
 
 ## Statistics
 
 - Pearson `r`: linear correlation. Larger absolute value means a straighter line.
 - Spearman `rho`: rank correlation. Larger absolute value means the ordering agrees more, even if the line is not clean.
-- Simple `R2`: Pearson `r` squared for one metric against the external score.
-- Release-controlled Elo: Pearson correlation after both external score and LLM Chess Elo are residualized on release month.
+- Raw Elo correlation: `corr(external_score, llm_chess_elo)` on the Elo sample.
+- Simple `R2`: Pearson `r` squared for one single metric against the external score.
 - OLS CV: repeated 5-fold cross-validation over 3 seeds. Features are selected inside each training fold from predeclared non-Elo chess metrics, then scored on held-out rows.
 
-## Current Signal
+### Release-Controlled Elo
+
+Raw Elo correlation uses two lists: external scores and LLM Chess Elo values. Release-controlled Elo uses the same final correlation operation, but first replaces both lists with release-date residuals.
+
+Starting from the Elo sample, keep rows with usable release-month metadata:
+
+1. Convert each model release date to a release-month value.
+2. Fit a linear model that predicts external score from release month.
+3. Subtract predicted external score from actual external score. This gives one external-score residual per row.
+4. Fit a separate linear model that predicts LLM Chess Elo from release month.
+5. Subtract predicted Elo from actual Elo. This gives one Elo residual per row.
+6. Compute Pearson correlation between the two residual lists.
 
-| Eval | Raw Elo | Release-controlled | Top chess metric | OLS CV |
-| --- | --- | --- | --- | --- |
-| Epoch ECI | Pearson `0.697`, Spearman `0.698`, `n=66` | Pearson `0.493`, `n=65` | `player_wins_percent`: Pearson `0.637`, Spearman `0.769`, `n=81` | `R2=0.381` vs mean baseline `-0.043`; rank Spearman `0.683` |
-| ARC-AGI-2 | Pearson `0.527`, Spearman `0.630`, `n=52` | Pearson `0.384`, `n=52` | `player_wins_percent`: Pearson `0.401`, Spearman `0.607`, `n=55` | `R2=-0.041` vs mean baseline `-0.044`; rank Spearman `0.485` |
+In formula form:
 
-Interpretation: ECI carries usable signal. ARC carries some rank-order signal, but its calibrated prediction result is basically baseline-level.
+```text
+raw_elo = corr(external_score, llm_chess_elo)
 
-## Trust Status
+external_residual = external_score - predict(external_score from release_month)
+elo_residual = llm_chess_elo - predict(llm_chess_elo from release_month)
 
-- Published artifacts currently reproduce: audit `reproducibility_status = pass`.
-- Coverage is not clean: audit `coverage_status = review-needed` because `169` mapping rows still need review.
-- Do not turn the correlations into capability claims until the high-impact mapping queue shrinks.
+release_controlled_elo = corr(external_residual, elo_residual)
+```
+
+The release-controlled value answers a stricter question: among models with comparable release timing, do models that beat their date-based external-score expectation also beat their date-based chess-Elo expectation? It removes only the linear release-month trend, not every possible time-related confounder.
+
+## Interpretation Boundaries
+
+- Correlation is association, not causation.
+- Raw Elo correlation can be inflated by chronology because newer models often improve across many benchmarks at once.
+- Release-controlled correlation is a timing check, not a full causal adjustment.
+- OLS CV is a prediction check; it can fail even when rank correlation is positive.
+- Mapping uncertainty is part of the result. Do not hide unresolved rows just because they are excluded from statistics.
 
 ## Review Order
 
-1. [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md): current findings and next actions.
-2. [results/cross_ref_summary.json](results/cross_ref_summary.json): machine-readable aggregate facts.
-3. [results/mapping_review.csv](results/mapping_review.csv): row-level mapping debt.
-4. [mappings/eci.csv](mappings/eci.csv) and [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv): identity source of truth used by the runner.
+1. Read [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) for the cross-eval bottom line and current caveats.
+2. Inspect [results/eci.html](results/eci.html) and [results/arc_agi_2.html](results/arc_agi_2.html) for per-eval detail.
+3. Check [results/eci_summary.json](results/eci_summary.json) and [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) when a claim needs machine-readable backing.
+4. Use [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), and the mapping CSVs to review row-level inclusion decisions.
+5. Use `mapping-review` or `audit` commands below for generated review surfaces that are not necessarily checked in.
 
 ## Commands
 
 From the repository root with `.venv` active:
 
 ```bash
+.venv/bin/python data/cross-ref/run_cross_ref.py eci
+.venv/bin/python data/cross-ref/run_cross_ref.py arc_agi_2
+.venv/bin/python data/cross-ref/run_cross_ref.py cross-eval
+.venv/bin/python data/cross-ref/run_cross_ref.py mapping-review
 .venv/bin/python data/cross-ref/run_cross_ref.py audit
 ```
 
-The default run writes temporary review outputs. Use `--publish` only after reviewing the output and intentionally updating checked-in artifacts.
+Default runs write scratch review outputs. Add explicit `--summary-output`, `--report-output`, `--csv-output`, or `--html-output` paths when you need a stable temporary artifact. Use `--publish` only after reviewing outputs and intentionally updating checked-in generated artifacts.
 
-Detailed agent-only command and artifact contracts live in the cross-ref skill references: [commands](../../.agents/skills/cross-ref-research/references/commands.md), [artifacts](../../.agents/skills/cross-ref-research/references/artifacts.md), [methodology](../../.agents/skills/cross-ref-research/references/methodology.md), and [mapping rules](../../.agents/skills/cross-ref-research/references/mapping-rules.md).
+Focused examples:
+
+```bash
+.venv/bin/python data/cross-ref/run_cross_ref.py mapping-review --status ambiguous --eval-id eci
+.venv/bin/python data/cross-ref/run_cross_ref.py rerun-diff eci
+.venv/bin/python data/cross-ref/run_cross_ref.py cross-eval --summary-output /tmp/cross_ref_summary.json
+```
 
 ## Layout
 
-- `evals/`: frozen external source snapshots and provenance notes.
-- `mappings/`: mapping CSVs read by the runner.
-- `mapping-research/`: evidence notes for mapping decisions.
-- `model-identity/`: generated LLM Chess model inventory.
-- `results/`: generated summaries, coverage, audit, and review surfaces.
-- `adapters/`, `framework/`, `run_cross_ref.py`: normalization, analysis, and publication code.
+- [evals](evals): frozen external source snapshots and provenance notes.
+- [mappings](mappings): mapping CSVs read by the runner.
+- [mapping-research](mapping-research): evidence notes for mapping decisions.
+- [model-identity](model-identity): generated LLM Chess model inventory.
+- [results](results): checked-in per-eval summaries, coverage files, and human reports.
+- [adapters](adapters), [framework](framework), and [run_cross_ref.py](run_cross_ref.py): normalization, analysis, and publication code.
