arc-agi updates, bs bench added

Author: Maxim Saplin

.agents/skills/cross-ref-research/SKILL.md

@@ -64,6 +64,29 @@ Refreshing an existing eval's source snapshot with newer upstream data has its o
 5. Separate the data effect from any code change. Checked-in `results/` baselines may predate the current code (signs: `llm_chess_inputs.data_quality` is null, or `prediction.ols.in_sample` is present). For a clean data-only diff, regenerate the baseline by running current code on the previous inputs — check the old snapshot and mapping out to `/tmp` — then diff the new run against that baseline.
 6. Read the test results in context. `tests/test_cross_ref.py` pins dataset-derived counts and correlations, so a refresh will move several of them; refresh those expectations as part of the change and confirm the structural assertions still hold.
 
+## Mistake Metrics (wrong actions / wrong moves / mistakes)
+
+Logs before `2025-03-16` underreported wrong actions and wrong moves, so the error/discipline
+metrics (`wrong_actions_per_1000moves`, `wrong_moves_per_1000moves`, `mistakes_per_1000moves`, and
+the `player_wrong_*` counts) are **excluded from analysis by default**. Do not quietly re-enable
+them.
+
+- Each model row in `elo_refined.csv` carries `min_game_date` (earliest game start). A model is
+  trustworthy for these metrics only when `min_game_date >= 2025-03-16` — then *every* game is
+  post-fix, so the published full-history value is already clean. The cutoff lives in one place:
+  `framework/data_quality.MISTAKE_STATS_TRUSTED_AFTER`.
+- To use these metrics in research, request clean mode: `run_cross_ref.py <eval> --mistake-stats
+  clean_only` (or `run_analysis(..., mistake_stats="clean_only")`). It drops every model with
+  `min_game_date` before the cutoff (or missing) and re-enables the repaired rate metrics for the
+  remaining sample. It is research-only and refuses `--publish`; published artifacts always run with
+  `mistake_stats="excluded"`.
+- Policy is **drop, never recover**: a model whose earliest game predates the cutoff is dropped
+  whole, even if it also has later games. We do not recompute individual models from a post-cutoff
+  subset.
+- `min_game_date` is produced by `data/get_refined_csv.py` during the normal build; it is the only
+  run-date provenance carried into the aggregate, so prefer it over model release dates for any
+  "was this tested after X" question.
+
 ## Optional Work Splitting
 
 For large mechanical row review or command-output verification, an agent may ask another agent to inspect a bounded slice. Keep the request mechanical, provide exact files and acceptance criteria, and re-check the answer against the source artifacts yourself. These helper notes are not cross-ref artifacts and are not required workflow.
@@ -76,4 +99,5 @@ For large mechanical row review or command-output verification, an agent may ask
 - ARC `COST (V3)` remains unresolved in located official sources; keep cost interpretation conservative.
 - Human baselines and benchmark-system rows stay visible in source and coverage outputs but are excluded from LLM Chess correlation samples.
 - Release-controlled correlations are lower than raw Elo correlations; do not present raw correlations as model-capability proof without the timing caveat.
-- If generated outputs differ after a rerun, inspect whether the difference comes from source, mapping, code, dependency behavior, or expected artifact metadata before publishing.
+- If generated outputs differ after a rerun, inspect whether the difference comes from source, mapping, code, dependency behavior, or expected artifact metadata before publishing.
+- The error/discipline metrics are excluded by default and only usable via `--mistake-stats clean_only`; see "Mistake Metrics" above. Never present them from a default run, and never publish a clean-only run.

data/cross-ref/CONSOLIDATED_REPORT.md

@@ -2,58 +2,127 @@
 
 ## Bottom Line
 
-Epoch ECI is the only supported external eval with a strong enough signal to treat as useful right now. ARC-AGI-2 has a positive Elo/rank relationship, but the prediction fit is baseline-level and the mapping ambiguity is much worse.
+Epoch ECI is still the only external eval with a strong enough signal to treat as useful. ARC-AGI-2
+has a moderate raw and rank relationship to LLM Chess Elo but its calibrated prediction is no better
+than the mean baseline. BullshitBench v2 is the weakest of the three: a modest rank relationship
+that mostly disappears once release timing is controlled.
 
-Do not sell this as broad benchmark validation. It is a two-eval comparison with unresolved identity debt.
+Do not sell this as broad benchmark validation. It is now a three-eval comparison with a large pile
+of unresolved identity debt, and the three evals disagree about which chess behaviors track external
+capability.
 
-Figures reflect the 2026-05-29 ECI refresh (overall Epoch index, 178 models) against the current `elo_refined.csv`. The ECI signal strengthened versus the prior snapshot, mainly from reconciling the mapping (e.g. `GPT-5.4` now `gpt-5.4-medium`) and adding four newly matched GPT-5.x models.
+Figures reflect the current published snapshots against the current `elo_refined.csv`: the
+2026-05-29 ECI refresh (178 models), the 2026-05 ARC-AGI-2 refresh (152 rows), and the newly added
+BullshitBench v2 leaderboard (162 model/reasoning rows, upstream commit `88e06ae`, 2026-05-29).
+
+BullshitBench measures one behavior — whether a model pushes back on a nonsense premise instead of
+playing along — on a 0-2 scale where higher is better. It is a deliberately different axis from a
+capability index like ECI, and the data show it: the strongest nonsense-detectors in the matched
+sample are mid-Elo Claude models, while the highest-Elo OpenAI/Gemini reasoning models sit mid-pack.
 
 ## Signal Table
 
 | Eval | Raw Elo | Release-controlled | Top chess metric | OLS CV |
 | --- | --- | --- | --- | --- |
-| Epoch ECI | Pearson `0.757`, Spearman `0.744`, `n=70` | Pearson `0.551`, `n=69` | `player_wins_percent`: Pearson `0.689`, Spearman `0.791`, `n=85` | `R2=0.527` vs mean baseline `-0.021`; rank Spearman `0.739` |
-| ARC-AGI-2 | Pearson `0.520`, Spearman `0.637`, `n=54` | Pearson `0.348`, `n=54` | `player_wins_percent`: Pearson `0.360`, Spearman `0.583`, `n=57` | `R2=-0.016` vs mean baseline `-0.042`; rank Spearman `0.494` |
+| Epoch ECI | Pearson `0.757`, Spearman `0.744`, `n=70` | Pearson `0.551`, `n=69` | `player_wins_percent`: Pearson `0.689`, Spearman `0.791`, `n=85` | `R2=0.527`; rank Spearman `0.739` |
+| ARC-AGI-2 | Pearson `0.584`, Spearman `0.660`, `n=57` | Pearson `0.412`, `n=57` | `average_game_cost`: Pearson `0.473`, Spearman `0.621`, `n=58` | `R2=-0.119`; rank Spearman `0.522` |
+| BullshitBench v2 | Pearson `0.303`, Spearman `0.480`, `n=55` | Pearson `0.118`, `n=54` | `average_game_cost`: Pearson `0.428`, Spearman `0.575`, `n=56` | `R2=0.090`; rank Spearman `0.472` |
 
 Interpretation:
 
-- ECI: usable relationship. The signal survives release-month control, and fold-local OLS beats the mean baseline.
-- ARC-AGI-2: weak relationship. Rank ordering is not empty, but calibrated prediction does not beat baseline in a meaningful way.
-- Shared signal: `player_wins_percent` is the strongest non-Elo chess metric for both evals.
+- ECI: usable relationship. The signal survives release-month control, and fold-local OLS predicts
+  the index well above a trivial baseline.
+- ARC-AGI-2: weak relationship under prediction. Raw and rank correlation are moderate, but the
+  calibrated OLS prediction is below the mean baseline (`R2=-0.119`), so treat ARC as a rank-order
+  signal only.
+- BullshitBench: weakest of the three. Rank ordering is positive (`Spearman 0.480`) but the linear
+  fit is low and the relationship mostly collapses under release-month control (`0.303` → `0.118`),
+  meaning most of the apparent association is chronology, not a model-capability link.
+- No shared headline metric anymore: `player_wins_percent` is the strongest non-Elo chess metric for
+  ECI, but for ARC and BullshitBench `average_game_cost` leads instead. For BullshitBench in
+  particular, `player_wins_percent` is essentially flat (Pearson `0.060`), so chess win rate does
+  not predict nonsense detection at all.
 
 ## Method In One Screen
 
 - Only mapped rows with status `accepted`, `alias`, or `variant-compatible` enter analysis.
-- `ambiguous`, `unmatched`, and `excluded` rows stay visible in coverage and mapping review, but do not enter correlations.
+- `ambiguous`, `unmatched`, and `excluded` rows stay visible in coverage and mapping review, but do
+  not enter correlations.
 - Multiple external rows for one LLM Chess player are deduped by keeping the highest external score.
+  For BullshitBench this collapses a model's reasoning-level rows to one model-level point.
 - Pearson `r` measures linear fit; Spearman `rho` measures rank-order fit.
 - Raw Elo correlates external score directly with LLM Chess Elo.
-- Release-controlled Elo first predicts each side from release month, subtracts predicted from actual on both sides, then correlates the two residual lists.
-- OLS CV uses repeated 5-fold cross-validation over 3 seeds. Feature selection happens inside each training fold, not on the full target sample.
+- Release-controlled Elo first predicts each side from release month, subtracts predicted from
+  actual on both sides, then correlates the two residual lists.
+- OLS CV uses repeated 5-fold cross-validation over 3 seeds.
+  Feature selection happens inside each training fold, not on the full target sample.
 
 ## Coverage Debt
 
-| Eval | Numeric rows | Accepted mappings | Metric sample | Elo sample | Unmatched external rows | High-impact unresolved |
+| Eval | Numeric rows | Accepted mappings | Metric sample | Elo sample | Unmatched/ambiguous rows | High-impact unresolved |
 | --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| Epoch ECI | 178 | 93 | 85 | 70 | 85 | 15 |
-| ARC-AGI-2 | 150 | 65 | 57 | 54 | 90 | 15 |
-
-The audit currently reports `reproducibility_status = pass`, `coverage_status = review-needed`, and `unresolved_row_count = 175`.
+| Epoch ECI | 178 | 93 | 85 | 70 | 85 / 0 | 15 |
+| ARC-AGI-2 | 152 | 71 | 60 | 57 | 24 / 54 | 15 |
+| BullshitBench v2 | 162 | 66 | 58 | 55 | 90 / 6 | 15 |
+
+The audit currently reports `reproducibility_status = pass`, `coverage_status = review-needed`, and
+`unresolved_row_count = 267` across the three external evals. All published per-eval artifacts
+reproduce cleanly (`evals_with_diff_count = 0`).
+
+### BullshitBench Mapping Shape
+
+The 162 leaderboard rows resolve to 6 `accepted`, 21 `alias`, and 39 `variant-compatible` mappings
+(66 mapped rows → 58 unique LLM Chess players, 55 in the Elo sample after dedupe), with 90
+`unmatched` and 6 `ambiguous`. Most unmatched rows are models with no LLM Chess counterpart (hosted
+Qwen 3.5/3.6/3.7, GLM-5-turbo/5.1/4.5, DeepSeek V4, Gemma 4, Nemotron Super/Nano-9B, MiMo, StepFun,
+Seed, ERNIE, Jamba, Prime Intellect, Arcee, GPT-5.5 Pro, and the `openrouter/*-alpha` stealth rows)
+or non-reasoning `none` rows for OpenAI families that exist in `elo_refined.csv` only at reasoning
+tiers. See `mapping-research/bullshit_bench.md` for the full effort-to-effort rule and the held-out
+`ambiguous` cases (grok-4.20-beta, an unspecified Claude 3.7 thinking budget, and two hosted-vs-quant
+local runs).
+
+## Exploratory: discipline beats strength (BullshitBench)
+
+This is a research lead, outside the governed numbers. Running BullshitBench in clean mode
+(`run_cross_ref.py bullshit_bench --mistake-stats clean_only` — drops every model whose earliest
+LLM Chess game predates the 2025-03-16 logging fix, leaving n=51 metric / 49 Elo) lets us use the
+normally-excluded error metrics. Rank correlation vs `avg_score`:
+
+| Chess metric | Spearman |
+| --- | ---: |
+| `wrong_actions_per_1000moves` | -0.58 |
+| `mistakes_per_1000moves` | -0.56 |
+| `games_interrupted_percent` | -0.42 |
+| LLM Chess Elo | +0.40 |
+| `player_wins_percent` | +0.03 |
+
+Sloppy chess (illegal actions, blunders, interrupted games) tracks *failing* BullshitBench better
+than raw strength does, and win rate is noise. It's a non-linear rank association only: a
+multi-factor OLS still doesn't beat the mean baseline (CV `R2=-0.13`). So nonsense detection looks
+like a carefulness/discipline trait, not a capability one — a lead, not a validated result.
 
 ## What Raises Signal
 
-1. Resolve high-impact mapping rows first: frontier GPT, Claude, Gemini Deep Think, DeepSeek, Qwen, Kimi, and GLM rows near the top of each leaderboard.
-2. Keep plain-model rows separate from reasoning effort, product tier, context-window, benchmark-system, and preview variants unless source evidence proves equivalence.
-3. Add more external evals only after their source score semantics and model identities are clear enough to avoid another ambiguity pile.
-4. Treat OLS prediction as secondary. The primary signal is still raw and release-controlled Elo correlation plus row-level mapping quality.
+1. Resolve high-impact mapping rows first: frontier GPT, Claude, Gemini, Grok, DeepSeek, Qwen, Kimi,
+   and GLM rows near the top of each leaderboard.
+2. Keep plain-model rows separate from reasoning effort, product tier, context-window,
+   benchmark-system, and preview variants unless source evidence proves equivalence.
+3. Treat BullshitBench as a behavioral contrast eval, not a capability eval. Its value is showing
+   where chess strength and epistemic pushback diverge, not validating Elo.
+4. Add more external evals only after their source score semantics and model identities are clear
+   enough to avoid another ambiguity pile.
+5. Treat OLS prediction as secondary. The primary signal is still raw and release-controlled Elo
+   correlation plus row-level mapping quality.
 
 ## Evidence Pointers
 
 - Methodology and workspace guide: [README.md](README.md)
 - ECI summary: [results/eci_summary.json](results/eci_summary.json)
 - ARC summary: [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json)
+- BullshitBench summary: [results/bullshit_bench_summary.json](results/bullshit_bench_summary.json)
 - ECI report: [results/eci.html](results/eci.html)
 - ARC report: [results/arc_agi_2.html](results/arc_agi_2.html)
-- ECI coverage: [results/eci_coverage.csv](results/eci_coverage.csv)
-- ARC coverage: [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv)
-- Mapping sources: [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv)
+- BullshitBench report: [results/bullshit_bench.html](results/bullshit_bench.html)
+- Coverage: [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), [results/bullshit_bench_coverage.csv](results/bullshit_bench_coverage.csv)
+- Mapping sources: [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv), [mappings/bullshit_bench.csv](mappings/bullshit_bench.csv)
+- Mapping rationale: [mapping-research/bullshit_bench.md](mapping-research/bullshit_bench.md)

data/cross-ref/README.md

@@ -11,13 +11,13 @@ This README is the operating guide. It should stay stable across result refreshe
 | Artifact | Role |
 | --- | --- |
 | [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) | Cross-eval findings, current caveats, and research next steps. |
-| [results/eci_summary.json](results/eci_summary.json), [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) | Machine-readable per-eval facts used for reporting. |
-| [results/eci.html](results/eci.html), [results/arc_agi_2.html](results/arc_agi_2.html) | Human-readable per-eval reports. |
-| [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv) | Row-level coverage and inclusion surfaces. |
-| [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv) | Runtime model-identity source of truth. |
+| [results/eci_summary.json](results/eci_summary.json), [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json), [results/bullshit_bench_summary.json](results/bullshit_bench_summary.json) | Machine-readable per-eval facts used for reporting. |
+| [results/eci.html](results/eci.html), [results/arc_agi_2.html](results/arc_agi_2.html), [results/bullshit_bench.html](results/bullshit_bench.html) | Human-readable per-eval reports. |
+| [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), [results/bullshit_bench_coverage.csv](results/bullshit_bench_coverage.csv) | Row-level coverage and inclusion surfaces. |
+| [mappings/eci.csv](mappings/eci.csv), [mappings/arc_agi_2.csv](mappings/arc_agi_2.csv), [mappings/bullshit_bench.csv](mappings/bullshit_bench.csv) | Runtime model-identity source of truth. |
 | [mapping-research](mapping-research) | Evidence notes behind mapping decisions. |
-| [evals/eci/SOURCE.md](evals/eci/SOURCE.md), [evals/arc-agi-2/SOURCE.md](evals/arc-agi-2/SOURCE.md) | External source provenance and score semantics. |
-| [evals/eci/epoch_eci_may_2026.csv](evals/eci/epoch_eci_may_2026.csv), [evals/arc-agi-2/arc-agi-2-apr-2026.csv](evals/arc-agi-2/arc-agi-2-apr-2026.csv) | Frozen external source snapshots. |
+| [evals/eci/SOURCE.md](evals/eci/SOURCE.md), [evals/arc-agi-2/SOURCE.md](evals/arc-agi-2/SOURCE.md), [evals/bullshit-bench/SOURCE.md](evals/bullshit-bench/SOURCE.md) | External source provenance and score semantics. |
+| [evals/eci/epoch_eci_may_2026.csv](evals/eci/epoch_eci_may_2026.csv), [evals/arc-agi-2/arc-agi-2-may-2026.csv](evals/arc-agi-2/arc-agi-2-may-2026.csv), [evals/bullshit-bench/bullshit_bench_v2_may_2026.csv](evals/bullshit-bench/bullshit_bench_v2_may_2026.csv) | Frozen external source snapshots. |
 
 ## Methodology
 
@@ -79,9 +79,9 @@ The release-controlled value answers a stricter question: among models with comp
 ## Review Order
 
 1. Read [CONSOLIDATED_REPORT.md](CONSOLIDATED_REPORT.md) for the cross-eval bottom line and current caveats.
-2. Inspect [results/eci.html](results/eci.html) and [results/arc_agi_2.html](results/arc_agi_2.html) for per-eval detail.
-3. Check [results/eci_summary.json](results/eci_summary.json) and [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json) when a claim needs machine-readable backing.
-4. Use [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), and the mapping CSVs to review row-level inclusion decisions.
+2. Inspect [results/eci.html](results/eci.html), [results/arc_agi_2.html](results/arc_agi_2.html), and [results/bullshit_bench.html](results/bullshit_bench.html) for per-eval detail.
+3. Check [results/eci_summary.json](results/eci_summary.json), [results/arc_agi_2_summary.json](results/arc_agi_2_summary.json), and [results/bullshit_bench_summary.json](results/bullshit_bench_summary.json) when a claim needs machine-readable backing.
+4. Use [results/eci_coverage.csv](results/eci_coverage.csv), [results/arc_agi_2_coverage.csv](results/arc_agi_2_coverage.csv), [results/bullshit_bench_coverage.csv](results/bullshit_bench_coverage.csv), and the mapping CSVs to review row-level inclusion decisions.
 5. Use `mapping-review` or `audit` commands below for generated review surfaces that are not necessarily checked in.
 
 ## Commands
@@ -91,6 +91,7 @@ From the repository root with `.venv` active:
 ```bash
 .venv/bin/python data/cross-ref/run_cross_ref.py eci
 .venv/bin/python data/cross-ref/run_cross_ref.py arc_agi_2
+.venv/bin/python data/cross-ref/run_cross_ref.py bullshit_bench
 .venv/bin/python data/cross-ref/run_cross_ref.py cross-eval
 .venv/bin/python data/cross-ref/run_cross_ref.py mapping-review
 .venv/bin/python data/cross-ref/run_cross_ref.py audit