Merge branch 'cross-ref'

Author: maxim-saplin

.agents/skills/cross-ref-research/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: cross-ref-research
+description: "Use when implementing, extending, auditing, or QAing LLM Chess data/cross-ref workflows: external eval source snapshots, conservative model mapping, run_cross_ref.py commands, generated results, consolidated reports, mapping review, cross-eval reports, reproducibility audits, and verification."
+---
+# Cross-Ref Research
+
+Use this skill for work under `data/cross-ref/`: adding or maintaining external evals, changing model mappings, regenerating published artifacts, auditing trust status, or updating the consolidated cross-eval report from generated facts.
+
+Do not use this skill for ad hoc leaderboard commentary, fuzzy one-off model matching, or claims that will not be backed by source snapshots, mapping rows, generated artifacts, and runtime checks.
+
+## Start Here
+
+1. Read `data/cross-ref/README.md` for the research workspace shape, artifact roles, and trust boundaries.
+2. Read `data/cross-ref/CONSOLIDATED_REPORT.md` when the task touches published findings or unresolved mapping caveats.
+3. Choose the narrow task path: output review, mapping row review, command execution, publication, or methodology change.
+4. Work from the repository root with `.venv` active. Non-publish commands default to review outputs outside `data/cross-ref/`; use `--publish` only when intentionally updating checked-in generated artifacts.
+
+## Procedure
+
+1. Anchor the task in the smallest controlled surface: source snapshot, mapping CSV, adapter, generated result, report section, or failing command.
+2. Preserve the workspace contract: source snapshots, mappings, mapping rationale, generated results, and code live in separate folders.
+3. Treat model identity as research. Accept a mapping only when evidence supports the exact model/configuration; keep uncertainty visible as `ambiguous`, `unmatched`, or `excluded`.
+4. Run the cheapest check that can disprove the current change before expanding scope. For mapping or adapter edits, that is usually a focused `run_cross_ref.py` command with explicit `/tmp` outputs or `tests/test_cross_ref.py`.
+5. Publish generated artifacts only through `data/cross-ref/run_cross_ref.py --publish`. Do not hand-edit files under `data/cross-ref/results/`.
+6. Update `data/cross-ref/CONSOLIDATED_REPORT.md` only from generated summaries and reports, then keep caveats explicit when unresolved rows constrain conclusions.
+7. Before reporting done, run the task-relevant verification commands and the requested final checks. Use the QA handoff skill when the change is implementation-heavy or affects published artifacts.
+
+## Examples
+
+Existing mapping correction:
+
+```text
+1. Inspect data/cross-ref/mappings/{eval_id}.csv and mapping-research notes.
+2. Update only rows with evidence; leave weak matches unresolved.
+3. Run the eval in review mode or to explicit `/tmp` outputs, then tests/test_cross_ref.py.
+4. If publishing, regenerate the eval artifacts, mapping_review.*, and cross-eval outputs as needed.
+```
+
+Existing output trust check:
+
+```text
+1. Run audit in default review mode while investigating, and add `--publish` only when refreshing checked-in audit outputs.
+2. Read audit status fields separately: reproducibility can pass while coverage remains review-needed.
+3. Trace headline claims through *_summary.json before editing narrative reports.
+```
+
+Adding another eval:
+
+```text
+1. Add evals/<eval-folder>/SOURCE.md with provenance, score meaning, columns, and caveats.
+2. Add adapter and runner registration.
+3. Add mappings/<eval_id>.csv with conservative statuses and rationale.
+4. Add focused tests, generate /tmp artifacts, review coverage, then publish through the runner.
+```
+
+## Optional Work Splitting
+
+For large mechanical row review or command-output verification, an agent may ask another agent to inspect a bounded slice. Keep the request mechanical, provide exact files and acceptance criteria, and re-check the answer against the source artifacts yourself. These helper notes are not cross-ref artifacts and are not required workflow.
+
+## Edge Cases
+
+- Python and pytest version strings are informational unless a command actually fails.
+- ARC `COST (V3)` remains unresolved in located official sources; keep cost interpretation conservative.
+- Human baselines and benchmark-system rows stay visible in source and coverage outputs but are excluded from LLM Chess correlation samples.
+- Release-controlled correlations are lower than raw Elo correlations; do not present raw correlations as model-capability proof without the timing caveat.
+- If generated outputs differ after a rerun, inspect whether the difference comes from source, mapping, code, dependency behavior, or expected artifact metadata before publishing.

.github/agents/gpt-l.md

@@ -0,0 +1,7 @@
+---
+name: gpt-l
+model: GPT-5.4 mini (copilot)
+description: General purpose fast model agent
+---
+
+General purpose agent

.github/agents/qa-1.md

@@ -1,6 +1,6 @@
 ---
 name: qa-1
-model: Claude Opus 4.7 (copilot)
+model: GPT-5.3-Codex (copilot)
 description: Use for QA
 ---
 

.github/agents/qa-2.md

@@ -1,6 +1,6 @@
 ---
 name: qa-2
-model: Gemini 3.1 Pro (Preview) (copilot)
+model: GPT-5.4 (copilot)
 description: Use for QA
 ---
 

.github/agents/qa.md

@@ -1,6 +1,6 @@
 ---
 name: qa
-model: Claude Opus 4.7 (copilot)
+model: GPT-5.4 (copilot)
 description: Use for QA
 ---
 

data/cross-ref/CONSOLIDATED_REPORT.md

@@ -0,0 +1,57 @@
+# Consolidated Cross-Ref Report
+
+## Bottom Line
+
+Epoch ECI is the only supported external eval with a strong enough signal to treat as useful right now. ARC-AGI-2 has a positive Elo/rank relationship, but the prediction fit is baseline-level and the mapping ambiguity is much worse.
+
+Do not sell this as broad benchmark validation. It is a two-eval comparison with unresolved identity debt.
+
+## Signal Table
+
+| Eval | Raw Elo | Release-controlled | Top chess metric | OLS CV |
+| --- | --- | --- | --- | --- |
+| Epoch ECI | Pearson `0.697`, Spearman `0.698`, `n=66` | Pearson `0.493`, `n=65` | `player_wins_percent`: Pearson `0.637`, Spearman `0.769`, `n=81` | `R2=0.381` vs mean baseline `-0.043`; rank Spearman `0.683` |
+| ARC-AGI-2 | Pearson `0.527`, Spearman `0.630`, `n=52` | Pearson `0.384`, `n=52` | `player_wins_percent`: Pearson `0.401`, Spearman `0.607`, `n=55` | `R2=-0.041` vs mean baseline `-0.044`; rank Spearman `0.485` |
+
+Interpretation:
+
+- ECI: usable relationship. The signal survives release-month control, and fold-local OLS beats the mean baseline.
+- ARC-AGI-2: weak relationship. Rank ordering is not empty, but calibrated prediction does not beat baseline in a meaningful way.
+- Shared signal: `player_wins_percent` is the strongest non-Elo chess metric for both evals.
+
+## Method In One Screen
+
+- Only mapped rows with status `accepted`, `alias`, or `variant-compatible` enter analysis.
+- `ambiguous`, `unmatched`, and `excluded` rows stay visible in coverage and mapping review, but do not enter correlations.
+- Multiple external rows for one LLM Chess player are deduped by keeping the highest external score.
+- Pearson `r` measures linear fit; Spearman `rho` measures rank-order fit.
+- Raw Elo correlates external score directly with LLM Chess Elo.
+- Release-controlled Elo first predicts each side from release month, subtracts predicted from actual on both sides, then correlates the two residual lists.
+- OLS CV uses repeated 5-fold cross-validation over 3 seeds. Feature selection happens inside each training fold, not on the full target sample.
+
+## Coverage Debt
+
+| Eval | Numeric rows | Accepted mappings | Metric sample | Elo sample | Unmatched external rows | High-impact unresolved |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Epoch ECI | 161 | 89 | 81 | 66 | 79 | 15 |
+| ARC-AGI-2 | 150 | 65 | 55 | 52 | 92 | 15 |
+
+The audit currently reports `reproducibility_status = pass`, `coverage_status = review-needed`, and `unresolved_row_count = 169`.
+
+## What Raises Signal
+
+1. Resolve high-impact mapping rows first: frontier GPT, Claude, Gemini Deep Think, DeepSeek, Qwen, Kimi, and GLM rows near the top of each leaderboard.
+2. Keep plain-model rows separate from reasoning effort, product tier, context-window, benchmark-system, and preview variants unless source evidence proves equivalence.
+3. Add more external evals only after their source score semantics and model identities are clear enough to avoid another ambiguity pile.
+4. Treat OLS prediction as secondary. The primary signal is still raw and release-controlled Elo correlation plus row-level mapping quality.
+
+## Evidence Pointers
+
+- Methodology and workspace guide: [README.md](README.md)
+- ECI summary: [results/eci_summary.json](results/eci_summary.json)
+- ARC summary: [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json)
+- ECI report: [results/eci.html](results/eci.html)
+- ARC report: [results/arc_agi_2.html](results/arc_agi_2.html)
+- ECI coverage: [results/eci_coverage.csv](results/eci_coverage.csv)
+- ARC coverage: [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv)
+- Mapping sources: [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv)

data/cross-ref/README.md

@@ -0,0 +1,116 @@
+# Cross-Ref Research Workspace
+
+## Purpose
+
+This workspace tests whether frozen external benchmark scores line up with LLM Chess outcomes after conservative model identity mapping.
+
+This README is the operating guide. It should stay stable across result refreshes: use it for layout, artifact roles, commands, and methodology. Use [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) and the per-eval result artifacts for current findings.
+
+## Artifact Map
+
+| Artifact | Role |
+| --- | --- |
+| [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) | Cross-eval findings, current caveats, and research next steps. |
+| [results/eci_summary.json](results/eci_summary.json), [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) | Machine-readable per-eval facts used for reporting. |
+| [results/eci.html](results/eci.html), [results/arc_agi_2.html](results/arc_agi_2.html) | Human-readable per-eval reports. |
+| [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv) | Row-level coverage and inclusion surfaces. |
+| [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv) | Runtime model-identity source of truth. |
+| [mapping-research](mapping-research) | Evidence notes behind mapping decisions. |
+| [evals/eci/SOURCE.md](evals/eci/SOURCE.md), [evals/arc-agi-2/SOURCE.md](evals/arc-agi-2/SOURCE.md) | External source provenance and score semantics. |
+| [evals/eci/epoch_eci_apr_2026.csv](evals/eci/epoch_eci_apr_2026.csv), [evals/arc-agi-2/arc-agi-2-apr-2026.csv](evals/arc-agi-2/arc-agi-2-apr-2026.csv) | Frozen external source snapshots. |
+
+## Methodology
+
+### Pipeline
+
+1. Parse each frozen external snapshot into normalized rows with one numeric target score.
+2. Map external model names to canonical LLM Chess player names through the mapping CSVs.
+3. Include only `accepted`, `alias`, and `variant-compatible` mappings in statistics.
+4. Keep `ambiguous`, `unmatched`, and `excluded` rows visible in coverage outputs, but out of correlations.
+5. Join included mappings to LLM Chess rows by canonical player name.
+6. If multiple external rows map to one chess player, keep the highest external score for that player.
+7. Compute non-Elo metric relationships on `metric_analysis_rows_max_dedupe`.
+8. Compute Elo relationships on `elo_analysis_rows_max_dedupe`, which requires non-null external score and LLM Chess Elo.
+
+The mapping step is intentionally conservative. A plausible name match is not enough when reasoning mode, product tier, context window, benchmark system, or preview status could change the identity of the model being evaluated.
+
+## Statistics
+
+- Pearson `r`: linear correlation. Larger absolute value means a straighter line.
+- Spearman `rho`: rank correlation. Larger absolute value means the ordering agrees more, even if the line is not clean.
+- Raw Elo correlation: `corr(external_score, llm_chess_elo)` on the Elo sample.
+- Simple `R2`: Pearson `r` squared for one single metric against the external score.
+- OLS CV: repeated 5-fold cross-validation over 3 seeds. Features are selected inside each training fold from predeclared non-Elo chess metrics, then scored on held-out rows.
+
+### Release-Controlled Elo
+
+Raw Elo correlation uses two lists: external scores and LLM Chess Elo values. Release-controlled Elo uses the same final correlation operation, but first replaces both lists with release-date residuals.
+
+Starting from the Elo sample, keep rows with usable release-month metadata:
+
+1. Convert each model release date to a release-month value.
+2. Fit a linear model that predicts external score from release month.
+3. Subtract predicted external score from actual external score. This gives one external-score residual per row.
+4. Fit a separate linear model that predicts LLM Chess Elo from release month.
+5. Subtract predicted Elo from actual Elo. This gives one Elo residual per row.
+6. Compute Pearson correlation between the two residual lists.
+
+In formula form:
+
+```text
+raw_elo = corr(external_score, llm_chess_elo)
+
+external_residual = external_score - predict(external_score from release_month)
+elo_residual = llm_chess_elo - predict(llm_chess_elo from release_month)
+
+release_controlled_elo = corr(external_residual, elo_residual)
+```
+
+The release-controlled value answers a stricter question: among models with comparable release timing, do models that beat their date-based external-score expectation also beat their date-based chess-Elo expectation? It removes only the linear release-month trend, not every possible time-related confounder.
+
+## Interpretation Boundaries
+
+- Correlation is association, not causation.
+- Raw Elo correlation can be inflated by chronology because newer models often improve across many benchmarks at once.
+- Release-controlled correlation is a timing check, not a full causal adjustment.
+- OLS CV is a prediction check; it can fail even when rank correlation is positive.
+- Mapping uncertainty is part of the result. Do not hide unresolved rows just because they are excluded from statistics.
+
+## Review Order
+
+1. Read [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) for the cross-eval bottom line and current caveats.
+2. Inspect [results/eci.html](results/eci.html) and [results/arc_agi_2.html](results/arc_agi_2.html) for per-eval detail.
+3. Check [results/eci_summary.json](results/eci_summary.json) and [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) when a claim needs machine-readable backing.
+4. Use [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), and the mapping CSVs to review row-level inclusion decisions.
+5. Use `mapping-review` or `audit` commands below for generated review surfaces that are not necessarily checked in.
+
+## Commands
+
+From the repository root with `.venv` active:
+
+```bash
+.venv/bin/python data/cross-ref/run_cross_ref.py eci
+.venv/bin/python data/cross-ref/run_cross_ref.py arc_agi_2
+.venv/bin/python data/cross-ref/run_cross_ref.py cross-eval
+.venv/bin/python data/cross-ref/run_cross_ref.py mapping-review
+.venv/bin/python data/cross-ref/run_cross_ref.py audit
+```
+
+Default runs write scratch review outputs. Add explicit `--summary-output`, `--report-output`, `--csv-output`, or `--html-output` paths when you need a stable temporary artifact. Use `--publish` only after reviewing outputs and intentionally updating checked-in generated artifacts.
+
+Focused examples:
+
+```bash
+.venv/bin/python data/cross-ref/run_cross_ref.py mapping-review --status ambiguous --eval-id eci
+.venv/bin/python data/cross-ref/run_cross_ref.py rerun-diff eci
+.venv/bin/python data/cross-ref/run_cross_ref.py cross-eval --summary-output /tmp/cross_ref_summary.json
+```
+
+## Layout
+
+- [evals](evals): frozen external source snapshots and provenance notes.
+- [mappings](mappings): mapping CSVs read by the runner.
+- [mapping-research](mapping-research): evidence notes for mapping decisions.
+- [model-identity](model-identity): generated LLM Chess model inventory.
+- [results](results): checked-in per-eval summaries, coverage files, and human reports.
+- [adapters](adapters), [framework](framework), and [run_cross_ref.py](run_cross_ref.py): normalization, analysis, and publication code.

data/cross-ref/adapters/__init__.py

@@ -0,0 +1 @@
+"""Adapters for external evaluation sources."""