003-transferability-proxy.md

003 — Transferability-proxy retrofit (does a cheap pre-fit signal predict the warm−cold sign?)

Status: NOT-TESTABLE (eval 2026-07-21). N_dec = 5 < 6 under the frozen decidability rule — too few decidable (world, seed) groups survive in the already-collected Gate 001/002 data to reject the 50%-coin null even at perfect sign accuracy. See "Result" below; the analysis stopped at the pre-registered NOT-TESTABLE floor, before computing sign accuracy, τ, or either control.

Status (original registration, kept for the record): REGISTERED (2026-07-21). Pre-registered BEFORE any proxy value is computed against the Gate 001/002 numbers. This is a re-analysis of already-collected data (Gates 001 and 002: four (world × arm) fits × three seeds, all on disk) — zero new ES compute. No new fit runs; the only new code computes a closed-form proxy from vectors and transition batches that already exist. Rung: ROADMAP short-horizon S1 (study-round candidate #1). Both prior gates STOPped/downgraded because the W1 world-model prior confers no reliable average advantage over cold fit at 32–128 transitions. This gate does not re-litigate that; it asks a cheaper, orthogonal question: before spending the ES on a transfer attempt, can a closed-form read of W1 against the target's held-out transitions predict whether that attempt will help or hurt (the sign of warm−cold)? A GO would license wiring the proxy as a pre-fit filter into the Phase-D transfer path (M2 Gauntlet warm-start selection); a STOP benches the "cheap transferability proxy" idea for this ladder.

The 2026-07-20 research round (docs/RESEARCH-NOTES.md, top; JOURNAL "Cross-world transfer SOTA study round") converged on this from four literatures: ML transferability estimation (LogME/Task2Vec/LEEP — a source→target score computed without training predicts transfer), curriculum CMDP source-task correlation, and biology's schema-congruence pre-check (van Kesteren's SLIMM — the sharp finding that partial congruence can actively hurt, not merely fail to help; a candidate explanation for Gate 001 seed-1 negative transfer). The falsifiable claim below is the monotone version of that literature; SLIMM's non-monotonicity is one way it refutes.

---

Hypothesis

A single, closed-form, ES-free proxy P — the raw-prior predictive fit of the W1 weights on the target world's held-out transition batch (score W1 as-is on the target, no fitting) — predicts the sign of the already-measured warm−cold margin across the decidable cells of the Gate 001/002 data, via the frozen monotone rule "predict warm ≥ cold iff P ≥ τ", at a sign-prediction accuracy that (i) rejects the 50% coin null by a one-sided exact binomial at α = 0.05, and (ii) exceeds both a shuffled-pairing null and a random-vector proxy. If the relationship is non-monotone (SLIMM: high congruence sometimes hurts), or P predicts sign no better than a coin, the hypothesis is refuted.

Data inventory (frozen from disk; a function of availability, not of outcomes)

The (proxy, outcome) pairs already exist; the proxy has simply never been computed. What is on disk:

• Gate 002 (PRIMARY — complete raw per-rep on disk): scratch/gate002_seed{0,1,2}.txt. Per (world ∈ {columns, room} × budget ∈ {32, 128} × seed ∈ {0,1,2}) cell: 5 CRN-paired per-rep RA64 cold/warm/scr/id and hence 5 paired per-rep MK = warm − cold; plus RAWPRIOR RA64 per (world, seed) and the W1 prior's own train-changed and held-out RA64 per seed. 12 cells; 60 paired per-rep MK values. RA64 is the K-step rollout ruler Route A proved is the real consumer.
• Gate 001 (SECONDARY / corroboration — published per-seed tables only, no raw per-rep on disk): docs/gates/001-wm-prior-transfer.md variance-reduced re-measure. Recoverable per-seed one-step changed-cell margins M: room {32,64,128} × {0,1,2} (9 cells) and columns {32} × {0,1,2} (3 cells) — ≤12 cells, medians only (no per-rep spread, so decidability is coarser here).
• Not on disk (benched, see below): per-iteration ES fitness traces (proxy b needs them) and the persisted cold/target-fit weight vectors (proxy c's cosine variant needs them). Neither can be had without new ES compute, which this gate forbids. They are documented not-computable, not silently dropped.

The proxy family (formulas frozen now)

Three candidates were named in the roadmap. Their computability from existing data differs; the gate rests on exactly one confirmatory proxy, with the others exploratory.

• (a) PRIMARY — raw-prior predictive fit P_a (closed-form, zero ES). Score the W1 weight vector w1 as-is on the target world's held-out transition batch — no warm/cold fit, no ES iteration. Under the K-step ruler this is exactly the RAWPRIOR RA64 already logged per (world, seed); its one-step analogue is a single held_out_score pass of w1 on the target held-out batch. Mechanically this is the "closed-form least-squares read of W1 against the target's held-out transitions" of the roadmap: the residual/accuracy of W1's own prediction on unseen target transitions, optionally with a closed-form scalar readout fit (LogME-style) that adds no ES. Registered limitation, stated up front: P_a ignores the training subset, so it is constant across budgets within a (world, seed) — it has 6-value resolution (2 worlds × 3 seeds) and cannot see the budget axis (the data-efficiency signature). The primary sign-prediction is therefore scored at (world, seed) resolution (see Metric).
• (b) EXPLORATORY — cold-fit ES fitness slope. The early-iteration slope of the cold arm's ES fitness (a learnability signal). Not computable from existing logs — no per-iteration traces were persisted; recomputing them is new ES compute (forbidden). Benched; included only if a trace is later found recoverable with zero new fits, and then exploratory-only.
• (c) EXPLORATORY — raw-prior congruence. L2/cosine distance between w1 and a target-fit reference. The cosine-to-cold-weights variant needs persisted fitted vectors (not on disk). A computable variant — congruence = P_a(w1) minus P_a of a norm-matched random vector on the same held-out batch — folds into control #2 below. Exploratory-only.

Confirmatory test = proxy (a) alone. (b)/(c) are exploratory; a pass on an exploratory proxy while (a) fails is at most PARTIAL, never GO (guards against multiple-comparison mining).

Metric + consumer

• Metric — sign-prediction accuracy of the frozen rule. For each decidable (world, seed) group (decidability rule below), the outcome sign is warm≥cold vs warm<cold; the prediction is sign(P_a − τ). The metric is the fraction of decidable groups whose sign the rule gets right, scored against a one-sided exact binomial with p = 0.5. Supporting (non-gating): Spearman ρ between P_a and the continuous pooled margin, with its (wide) CI, reported for texture only.
• Threshold τ (frozen now, before any Gate 001/002 number is read for this analysis): τ = median(P_a over the 6 Gate-002 (world,seed) RAWPRIOR values). Using the internal median as the split makes the rule scale-free and self-defining — it is not hand-placed to separate the known signs. The rule direction is the literature prior: higher P_a ⇒ predict warm ≥ cold (transfer helps when the raw prior already fits the target). The opposite direction is not a free second bet — if only the reversed rule works, that is the SLIMM non-monotone refutation, booked as STOP/PARTIAL, not re-labelled a GO.
• Consumer. No code reads a number yet — the consumer is the S1 GO/STOP decision itself: a GO green-lights building P_a as a pre-fit filter on the warm-start choice in the Phase-D transfer path (the fit_operator warm-start buffer in src/esper_evolution.mojo, selected per target world in the M2 Gauntlet / src/ued.mojo) — "check the proxy before spending ES on a warm start." A STOP means no cheap proxy predicts transfer sign on this data; do not build the filter; bench candidate #1. The decision reads sign-accuracy + the two controls, nothing else.

Analysis unit, decidability, and N (frozen)

• Unit — (world, seed) group for the primary test (forced by P_a's budget-blindness). A group's outcome sign is pooled over its budget cells' paired per-rep margins.
• Decidability (defines the eval set from outcomes, which are already known — this is legitimate; it does not touch the proxy): a per-rep margin counts as a win/loss only if |MK| > δ, δ = 0.005 (a numeric-noise floor; exact ties do not count as wins). A (world, seed) group is DECIDABLE and POSITIVE if ≥ 70% of its non-tie per-rep margins are positive, DECIDABLE and NEGATIVE if ≥ 70% are negative, and UNDECIDED (excluded) otherwise or if fewer than 3 non-tie reps exist. δ, the 70% rule, and the 3-rep minimum are frozen here and may not be moved after seeing accuracy.
• N. Let N_dec = number of decidable (world, seed) groups pooled across Gate 002 (≤6) and the Gate 001 corroboration set (≤6, medians-only decidability: |M_median| > 0.05). The known fragility, stated now: Gate 002's margins are heavily concentrated at exactly 0.0 (most cells are ties), so N_dec is expected to be small — this is the honest small-power regime, and the NOT-TESTABLE floor below is a first-class outcome, not a failure.

GO / STOP / PARTIAL / NOT-TESTABLE thresholds (committed now)

The bar is a self-adjusting exact binomial so it stays honest at whatever tiny N_dec the data yields (e.g. N_dec = 6 requires 6/6; N_dec = 8 requires ≥7/8; N_dec = 10 requires ≥9/10 — all one-sided p ≤ 0.05 under the 50% null).

NOT-TESTABLE (check first): N_dec < 6. Too few decidable groups to reject a coin at α = 0.05 even with perfect accuracy → the proxy question cannot be answered on this dataset. Booked as a documented "underpowered — needs fresh cells with robustly-nonzero margins," not as a pass and not as a refutation. (Given the near-degenerate outcome distribution, this is the likely verdict.)

GO — all three must hold (only evaluated if testable):

1. Signal: the frozen rule's sign-accuracy on the N_dec decidable groups rejects the 50% null at one-sided exact-binomial p ≤ 0.05.
2. Robustness (the re-analysis analogue of ≥2–3 independent draws): leave-one-seed-out — removing any single seed's groups, the point-estimate accuracy stays > 0.5 (no lone (world, seed) carries the result). No single group flip may cross the binomial bar.
3. Both controls sit at chance: the shuffled-pairing null's 95th-percentile accuracy is below the real accuracy, AND the random-vector proxy's accuracy does not reject the coin null.

STOP — any one triggers (proxy idea refuted for this ladder):

1. No better than a coin: sign-accuracy ≤ 0.5 on the decidable groups (proxy carries no sign information; candidate #1 benched).
2. Control collapse: the random-vector proxy predicts sign as well as w1 (rejects the coin null itself), OR the real accuracy fails to exceed the shuffle null's 95th percentile — any apparent signal is a relabeling artefact, not W1's transferable structure (parallel to Gate 001/002's scrambled-W1 control).
3. Wrong-direction monotonicity (SLIMM): only the reversed rule (higher P_a ⇒ warm worse) clears the binomial. Recorded as evidence for congruence-hurts, and STOP for the registered (helps-when-congruent) hypothesis; may license an insurance-shaped follow-up, never the filter.

PARTIAL — real-but-underpowered or exploratory-only signal without a STOP, e.g.:

• Point-estimate accuracy > 0.5 and beats the shuffle null, but misses the binomial bar for lack of N_dec — suggestive; needs fresh cells, not a filter.
• The primary proxy (a) fails but an exploratory proxy (b/c, if it became computable with zero new compute) passes — booked with a "confirm on fresh data first" caveat, never a GO.
• Gate 002 (primary) clears but the Gate 001 corroboration disagrees (or vice-versa) — the sign signal is objective-specific, not a general transferability read.

Seeds / independent-draw plan

This is a re-analysis, so the "independent draws" are the seeds already in the data: Gate 002 seeds 0, 1, 2 (both worlds) and Gate 001 seeds 0, 1, 2 (room) / 0, 1, 2 (columns re-measure). A single-seed fluke is refused two ways: the GO #2 leave-one-seed-out jackknife, and the requirement that decidability be met by ≥ 70% of per-rep draws within a group (not one lucky rep). seed(0)-only would be repeatability, not robustness — hence the cross-seed jackknife is a GO bar, not a footnote.

Refuting control (the load-bearing field)

Two controls, both must produce a "no" under the null, or the run is broken rather than blessed:

1. Shuffled pairing (permutation null). Randomly permute the assignment of P_a values to outcome groups, ≥ 10,000 shuffles; recompute sign-accuracy each. The real proxy must exceed the 95th percentile of this null. A proxy that carries genuine sign information beats a random relabeling; one that doesn't, can't. (Frozen: seed for the shuffle RNG = 0; α = 0.05 one-sided.)
2. Random-vector proxy (the scrambled-W1 analogue). Recompute P_a using a norm-matched random vector (Fisher-Yates permutation of w1, seed-fixed, never hand-chosen) in place of w1, scored on the same held-out batches. Its sign-accuracy must sit at chance. If a random vector predicts the sign as well as the real W1 prior, the proxy is not reading W1's transferable structure — exactly the failure Gate 001/002's scrambled arm was built to catch. This is the single most important check: it is the only thing separating "P_a reads real cross-world congruence" from "P_a correlates with per-world difficulty that any vector's raw score would also track."

If neither control can yield a "no" (e.g. every vector scores identically because the held-out floor saturates the metric), the hypothesis is not testable on this data — declare NOT-TESTABLE, do not manufacture a GO.

Anti-stone-soup clause

Because the outcomes are already public (published in Gates 001/002), the guarantee is procedural and total: the git commit registering this file — with the exact P_a formula, the direction, τ and its median definition, δ / the 70% rule / the 3-rep minimum, the (world, seed) unit, the N_dec floor, the binomial α, the primary-vs-exploratory split, and both controls — must predate the commit that first computes any proxy value against the Gate 001/002 numbers. The Result block must restate τ and the rule verbatim from this file. Any of the following makes a later "pass" a documented negative, not a GO:

• choosing or nudging τ (or its definition) after seeing which split separates the known signs — the median-of-P_a definition is frozen;
• moving δ, the 70% decidability threshold, the 3-rep minimum, or the |M_median|>0.05 Gate-001 rule to include/exclude the cells that make accuracy look good — the eval set is frozen by rule, not hand-picked;
• switching the analysis unit (rep vs cell vs (world,seed) group) to whichever gives significance, or pooling/splitting Gate 001 and Gate 002 to whichever reads better than each alone;
• adding per-world or per-budget offsets/normalizations to P_a, or fitting more than a single closed-form scalar readout (any ES/gradient tuning of the proxy = new compute and scaffolding);
• promoting an exploratory proxy (b/c) to "primary" post hoc, or reporting the reversed-direction rule as a GO instead of the SLIMM refutation it is;
• dropping a seed, world, or group the proxy gets wrong; loosening to a two-sided or larger α after seeing N_dec; hand-choosing the control permutation to be gentle;
• reporting the supporting Spearman ρ as if it were the gated metric (the gate is on sign accuracy vs a coin, with controls — a correlation that merely tracks per-world difficulty is a proxy for the wrong thing).

Mechanism reuse only: RAWPRIOR / held_out_score scoring of w1 (already in src/world_model.mojo), the transition batches and w1 vectors already on disk. The sole new code is the measurement-only proxy computation and the two controls — no new learning machinery, no new ES fit.

Result — NOT-TESTABLE (eval 2026-07-21)

New measurement produced. tests/test_gate003_proxy.mojo (skip-tier, reuses Gate 002's exact RNG prefix through the b=32 scr permutation, zero new ES fit) reproduced Gate 002's logged RAWPRIOR values bit-for-bit at all 3 seeds (sanity check passed unconditionally — confirms w1 and scr are the true already-used vectors) and produced the random-vector control's raw score:

seed	world	RAWPRIOR (P_a, W1)	RAWPRIOR_SCR (random-vector control)
0	columns	0.17721356	0.100618474
0	room	0.09375	0.05013021
1	columns	0.2544178	0.025292968
1	room	0.12363279	0.08320313
2	columns	0.24518234	0.15566407
2	room	0.08984375	0.0087890625

τ (median of the six P_a values, frozen definition) = 0.150423175. These numbers are recorded but not consumed — the NOT-TESTABLE floor triggers first (frozen rule: check this before computing sign accuracy), so no sign rule, τ-split, or control comparison was scored. They remain on disk for a future gate that adds seeds/worlds and reopens this question with more power.

Decidability computation (frozen rule: |MK| > δ=0.005 per rep, group is DECIDABLE iff ≥70% of its non-tie reps agree and ≥3 non-tie reps exist).

Gate 002 (scratch/gate002_seed{0,1,2}.txt, pooling MK over both budgets' 5 reps = 10 per group):

(world, seed)	non-tie reps	sign split	verdict
columns, 0	2	2 neg	UNDECIDED (< 3 non-tie)
columns, 1	0	—	UNDECIDED (all ties)
columns, 2	6	6 neg	DECIDABLE NEGATIVE
room, 0	8	8 neg	DECIDABLE NEGATIVE
room, 1	1	1 neg	UNDECIDED (< 3 non-tie)
room, 2	2	2 neg	UNDECIDED (< 3 non-tie)

→ 2 decidable groups.

Gate 001 corroboration set (docs/gates/001-wm-prior-transfer.md re-measure table, medians-only rule |M_median| > 0.05 per budget cell, group verdict = majority sign of its decidable cells):

(world, seed)	cells (M32/M64/M128)	decidable cells	verdict
room, 0	+0.012 / +0.091 / −0.071	2 (split +/−)	UNDECIDED (no majority)
room, 1	−0.024 / −0.002 / −0.029	0	UNDECIDED (none clear 0.05)
room, 2	+0.732 / +0.226 / +0.434	3, all +	DECIDABLE POSITIVE
columns, 0	+0.015 (B32 only)	0	UNDECIDED
columns, 1	−0.117 (B32 only)	1, −	DECIDABLE NEGATIVE
columns, 2	−0.057 (B32 only)	1, −	DECIDABLE NEGATIVE

→ 3 decidable groups.

N_dec = 2 + 3 = 5. Under the pre-registered NOT-TESTABLE clause (N_dec < 6), this is decisive on its own: even if every one of the 5 decidable groups' signs were predicted correctly by P_a, 5/5 does not clear the exact-binomial α=0.05 bar the gate itself requires reaching 6 observations to test (N_dec=6 needs 6/6). The two most plausible alternate readings of the Gate-001 single-budget columns cells (counting them as decidable at n=1, as scored above; or disqualifying single-cell groups as structurally underpowered, which would drop N_dec to 3) both land below 6 — the verdict is robust to this interpretive edge, not contingent on it.

Why this happened, matching the gate's own pre-registered expectation: Gate 002's margins are overwhelmingly exact ties at 0.0 (most cold/warm pairs converge to the identical fitted vector at this budget/objective) — precisely the fragility the gate's design section flagged before looking. Gate 001's medians-only cells add little power because most sit inside the ±0.05 noise band the gate itself found in the 2026-07-19 variance-reduced re-measure.

Verdict: NOT-TESTABLE, not STOP. This is a documented power failure of the available data, not a refutation of the proxy hypothesis — P_a, τ, and the random-vector control are computed and banked for reuse the moment a future rung (more seeds, more worlds, e.g. the Gauntlet's M1/M2 world family) grows N_dec past 6. No claim is made about whether P_a predicts transfer sign; none can be made honestly at this N. No STOP action follows — S1's candidate proxy is not benched, only deferred pending more decidable data; unlike Gates 001/002, this does not close the transferability- proxy question, it establishes that answering it needs a wider data base than Gates 001/002 alone provide.

Anti-stone-soup check: no threshold, rule, or analysis unit was altered after computing the groups above; the NOT-TESTABLE floor, δ, the 70% rule, and the |M_median|>0.05 rule are exactly as frozen at registration. The two named degrees of freedom in applying the Gate-001 rule to single-budget columns cells are disclosed above rather than silently resolved, and both readings agree on the outcome.