experiments: offline->downstream correlation, CPU proof-of-concept (keystone)

[Denis-hamon]

Tier 2.1.b/c (CPU slice of the keystone). You asked to lancer (a) — run the CPU part to get a first real offline→downstream correlation without the GPU. Here it is.

The question

Does a cheap, planner-free offline metric predict downstream planning success? The learned MLP arms on DMC sit at the planning floor (success 0 regardless of prediction quality — the dissociation the paper reports), so they give no downstream spread. The fast maze toy is the CPU-feasible setting where under-training the dynamics genuinely degrades planning, so success varies (0.0 → 0.30–0.60 → 1.0).

What it adds

• maze_quality_sweep.py (torch, in experiments/): 24 cells (epochs × seeds), three planner-free offline metrics vs the true maze dynamics — M1 one-step mismatch (naive), M2 k-step divergence (compounding), M3 action-ranking agreement (decision-aware) — plus downstream success_rate.
• correlate.py (stdlib): per-metric rank correlation + bootstrap CIs, plus a fair common-subset (equal-n) block and a comparability_note. Reuses the merged primitives (PR feat(metrics): stdlib rank-correlation primitives (Spearman, Kendall, bootstrap CI) #54).
• Committed bundles, README, tests.

Result (honest)

All three metrics predict success and clear zero. On the fair common subset (n=17) they are essentially equal: M1 −0.90, M2 −0.91, M3 +0.90 — the naive metric is not inferior to the decision-aware one here. That's the expected prediction ≈ decision signature of a small deterministic env, and it validates the pipeline end-to-end.

Not a claim of M3 superiority. The per-metric M3 (n=17) vs M1 (n=24) gap is an artifact of M3's smaller/easier subset (M3 is undefined for action-blind failures), not a real advantage — stated in the README and the comparability_note. The discriminating M1-vs-M3 test needs Stage 2 (DMC/TD-MPC2 cells, GPU), where capable models span non-floor downstream success.

Verification

• 239 tests pass; correlate.py stdlib-only; torch confined to the sweep; bundles are strict-valid JSON (M3 undefined → null, not NaN).
• Adversarial review GO after one round. The first round (NO-GO) caught a real bug: the n=17-vs-24 comparison manufactured a false "M3 strongest" headline — fixed with the equal-n common-subset, which vindicates the prediction≈decision caveat. Also fixed invalid bare-NaN JSON and a mislabeled comment. Caveats documented (M1/M2 rank-redundant ρ≈0.98; near-bimodal success; M2 single-sequence).

No new git tag. Paper section deferred to Stage 2 (this is an experiments/ artifact + README).

🤖 Generated with Claude Code