004-cbr-retain.md

004 — CBR retain: does a repertoire retained from TWO prior worlds retrieve into a THIRD better than one retained from ONE?

Status: STOP (2026-07-21). Measured via tests/test_cbr_retain.mojo, full 3-seed x 2-world run (seeds 0/1 primary AND rule, seed 2 tiebreak), scored strictly against the bars below — none moved after the number was seen.

Verdict rationale. On the PRIMARY triple {shelves,columns}→room (post-48-goal- escalation cells; all three room cells escalated past the 8-decidable floor):

seed	R1 (worse ONE-SRC/TWO-SRC)	Rc (RAND-MATCHED/TWO-SRC)	decidable
0	1.0	1.0	11/48
1	1.0	1.0	14/48
2 (tiebreak)	1.0	1.0	10/48

All three STOP triggers fire, at every seed: STOP #1 (R1 ≤ 1.0 — two real sources no better than one), STOP #2, the refuting control (Rc ≤ 1.05 — RAND-MATCHED, same count with scrambled/re-rolled fillers and zero retained structure, ties TWO-SRC exactly), and STOP #3 (precondition-unmet: advantage(room) − advantage(scatter) = 0.0 − 0.0 = 0.0 < +0.05 — scatter shows the same flat R1≈Rc≈1.0, so there is no room-specific recombination signal to distinguish "wrong world family" from "no signal at all"). G4 (cold cross-arm confound check) passed cleanly everywhere (ratio 1.0, all cells). No scaffolding: arms, constants, and goal counts ran exactly as frozen; the one implementation deviation (a goal generator checked disjoint against all five arm maps, not one) is the faithful realization of the gate's own "disjoint from every source repertoire" clause, mechanism-identical to gen_family_s.

Load-bearing addendum: NF = 0.0 in every (world, seed) cell — the full-budget few-shot fit converges to the same optimum regardless of fit-RNG seed almost everywhere, so most goals aren't even decidable (11-14/48 on room, 16-18/24 on scatter). This is the same "few-shot fit washes seed advantage out" phenomenon Gate 003/test_trial_select.mojo flagged, now reproduced under retain. Candidate #3 (write-everything retain, cold-solve- then-retain, no target-conditioning) is benched for the M2 Gauntlet warm arm; this walls family does not license wiring retain-write into it.

---

Pre-registered BEFORE any two-source vs one-source number is measured. The world triple, the retain policy, the size-matched refuting control, the decidability floor, and every GO/STOP/PARTIAL bar are frozen in this commit and may not move after a number is seen. Rung: ROADMAP short-horizon S2 — CBR retain gate (2026-07-20 study-round candidate #3). Route C (src/transfer.mojo, tests/test_trial_select.mojo) already runs CBR's retrieve (BC-kNN, EliteMap.nearest/nearest_k) → reuse (few-shot fit seeded from the winner, fit_operator[SandboxPolicyMemory]) → revise (real-trial argmin over the pool, pool_trials' 64-tick rollouts) stages. It has never run retain: nothing writes a target-world outcome back into a shared cross-world repertoire that a later world's retrieval draws on. This gate tests the retain step in the cheapest form the survey licenses — build a shared repertoire from two prior source worlds, and ask whether it retrieves into a held-out third world better than a repertoire built from a single source. A GO green-lights wiring a retain-write into the M2 Gauntlet warm arm (ROADMAP: "warm = retain + retrieve + ITE-select"); a STOP either benches candidate #3 or, via the frozen scatter contrast below, attributes the null to this world family failing Kashtan & Alon's precondition (deferring retain to M1/M2's built-for-purpose family) — the two are pre-distinguished so a null cannot be spun after the fact.

Grounding: docs/RESEARCH-NOTES.md §2 (Aamodt & Plaza 1994's retrieve→reuse→revise→retain cycle; the 2025 survey's utility-gated retain — novelty × effectiveness × generalizability, "not keep everything"). The Esper-mapping paragraph names this exact lever: "Retain is the cheapest of the two: measure whether a repertoire built from two prior worlds retrieves better into a third than a repertoire built from one."

---

Hypothesis

A shared repertoire retained from two source worlds ({shelves, columns}, merged by inserting every source elite into one EliteMap) retrieves and few-shot-transfers into a held-out third world (room) at a smaller held-out real-trial distance than a repertoire retained from either single source alone, AND this advantage survives a size-matched control (a repertoire of the same total elite count whose second-source half is random-assembled, not retained), AND it is larger on the recombination target (room) than on a non-recombination target (scatter). If two real sources beat one only as much as an equal-count random repertoire does, or if the advantage is no larger on room than on scatter, the retain hypothesis (as transfer of retained cross-world structure) is refuted — what helped, if anything, was candidate count / generic BC coverage, not retained structure.

Chosen world triple (frozen, with justification)

The four worlds are the existing walls family (gen_walls_layout, src/sandbox.mojo), reused unchanged — inventing worlds is M1's job, not S2's. Each isolates one thing the open world lacks: shelves = horizontal wall runs (mid-air support), columns = vertical dividers with door gaps (corridors/detours), room = a closed rectangle with one door (a closable region), scatter = ~14 irregular single-cell obstacles (no structured geometry).

• PRIMARY triple — source {shelves, columns} → held-out target room. Justification (a falsifiable shared-structure claim, not arbitrary): room's boundary geometry is, by construction, horizontal wall runs (rows 4 and 12 — the shelves ingredient) plus vertical wall runs with a door gap (col 4 solid, col 11 with a door — the columns ingredient). Its navigation sub-problems (settle blocks onto a horizontal ledge; route the avatar through a gap in a vertical divider) are the two the source worlds each isolate. This is the closest thing to a recombination target the existing family affords.
• CONTRAST triple — source {shelves, columns} → held-out target scatter. scatter's random single-cell obstacles are not a recombination of horizontal-run + vertical-divider geometry. The modularity hypothesis predicts advantage(room) > advantage(scatter); a two-source advantage that is equal on scatter is generic coverage, not shared structure. This differential is a first-class GO condition, not a footnote.

Honest precondition flag (load-bearing, pre-registered). Kashtan & Alon's precondition is modularly varying goals — the goal must decompose into world-specific sub-goals that recombine. This family varies obstacle geometry modularly-ish, but the goal is always the same shape (reach an end-state behaviour characterization); it was not built to decompose the goal into recombinable sub-goals (S1/Gate-003 already noted these worlds share topology-blocking mechanics but weren't built to share recombined sub-goals). Moreover room's "closable region" (containment) is a semantic not present in either source — so even the geometric recombination claim is partial. Consequence, committed now: a null on room is informative, not merely a miss — via the scatter contrast and STOP #3 it is booked as "this family does not satisfy the retain precondition; defer to M1/M2's built-for-purpose family," distinct from "retain-in-principle is refuted." A GO here would be a genuinely surprising positive given the weak precondition.

Retain policy (frozen — the single biggest gaming surface)

Write-everything through EliteMap.insert. Zero new free parameters. For each elite in a source world's built repertoire, call shared.insert(key, settle_t, w, bc) on the shared map; key collisions are resolved by the map's pre-existing, non-tunable fill-or-improve rule (keep the strictly smaller settle tick — src/map_elites.mojo:117). No novelty/effectiveness/ generalizability threshold is introduced.

Justification (why not the survey's utility-gated retain): utility-gating adds thresholds on novelty × effectiveness × generalizability — exactly the free knobs a later run could tune to manufacture a two-source win. For the first retain test the honest, non-gameable policy is the one with no free parameters, so the two-source-vs-one comparison is a clean function of source diversity alone. The EliteMap already applies a minimal, fixed dedup/quality rule (smaller settle wins) that is not a per-experiment knob. Utility-gated retain is deferred to a follow-up gate that only opens if write-everything shows signal — you do not tune a gate you have not yet earned. Naming or adding any utility threshold in the run = scaffolding (anti-stone-soup below).

Retain source, frozen. Each source world's repertoire is built by the locked me_emitter_run (constants BUILD_BUDGET/BUILD_RESEED/BUILD_N/BUILD_ALPHA/BUILD_SIGMA/INIT_SCALE from src/transfer.mojo, byte-identical across every arm/world/seed). "Retain from world X" = merge world X's built repertoire into the shared map by the insert loop above. The source worlds are solved/built without any knowledge of the target world — no target-conditioned retain. This is the clean cold-solve-then-retain registration; the richer compounding variant (world B retrieves from the A-so-far map while being built) introduces an ordering confound and is a different hypothesis, explicitly benched here (using it to boost = undocumented scaffolding).

Arms (frozen)

All arms retrieve/reuse/revise into the target world's held-out goals through the unchanged run_family_select / pool_trials; the only thing that differs is the contents of the shared EliteMap they read.

• TWO-SRC (treatment): shared map = retain(shelves) ⊕ retain(columns). Count ≈ N_AB.
• ONE-SRC-A / ONE-SRC-B (diversity baseline): shared map = retain(shelves) only (N_A), and separately retain(columns) only (N_B). GO must beat both.
• RAND-MATCHED (primary refuting control): shared map = retain(shelves) + random-assembled fillers to reach exactly N_AB. A filler is a weight-scrambled copy of a shelves elite (Fisher-Yates permutation of its POLICY_DIM vector, seed-fixed, never hand-chosen) re-rolled in the target-neutral open world to record its true resulting key+bc, so it is a valid map entry carrying no retained structure. This holds source A fixed and count fixed, replacing only B's real retained structure with equal-count random coverage. It is the direct adaptation of Route C's scrambled-elite / uniform-pool control to the retain question.
• RAND2X (secondary robustness control): shared map = N_AB elites from 2×BUILD_BUDGET of exploration in the open world (build-compute matched to TWO-SRC's two-world build), no walls structure. Isolates "two worlds of retained skill" from "two worlds' worth of exploration budget."
• cold (invariance sanity, ignores the map): zero-seed few-shot fit — must be statistically identical across every arm (it never reads the repertoire). A "retain win" that also moves cold is an artifact.

Metric + consumer

• Metric — the EXACT Route C held-out real-trial distance, no new scoring machinery. Per held-out goal g in the target world, d_ite[g] (and corroborating d_near[g]) as written by run_family_select / pool_trials (src/transfer.mojo): the negative real-trial rollout fitness (lower = closer) of the retrieved-and-refit policy against the goal BC, on a 64-tick deterministic rollout. Aggregated as the median over goals of the paired per-goal ratio d(comparator)/d(TWO-SRC) (Route A/C discipline — median of ratios, never mean; ratio > 1 ⇒ TWO-SRC closer), with the same EPS clamp Route C uses for exact reaches.
• Held-out currency (uncheatable by construction). Goals are gen_family_s-synthesized in the target world with keys disjoint from every source repertoire (the EliteMap.contains filter, leaks == 0 asserted as in Route C's G0). Retrieval can never return the answer; the few-shot fit must close a real gap. The target world's goals never enter any source map.
• Consumer. No proxy — the number the decision reads is the same held-out real-trial distance Route C's own consumer reads. The GO/STOP decision green-lights (or benches) building the retain-write into the M2 Gauntlet warm arm: a shared cross-world EliteMap accumulated by retain-writes, from which the Gauntlet's warm (retain+retrieve+ITE-select) arm retrieves (src/ued.mojo / the M2 driver). A GO licenses that wiring; a STOP either benches candidate #3 or defers it to M1/M2's family per STOP #3.

Decidability, N floor, and the escape hatch (frozen — the anti-Gate-003 clause)

Gate 003 died at N_dec = 5 < 6: a re-analysis of fixed data could not grow N. This gate generates its own data, so the trap is dodged by a pre-registered escape hatch, not hoped away.

• Noise floor NF (self-calibrating, not hand-placed). The cold arm is fit twice per goal with different fit-RNG (cold_a, cold_b); NF = median_g |d(cold_a,g) − d(cold_b,g)| — the metric's own per-goal test–retest spread. NF is defined by the data, not chosen.
• Decidable goal. A goal counts toward the median/binomial iff the paired |d(ONE-SRC,g) − d(TWO-SRC,g)| > NF (the difference clears the instrument's own noise).
• N floor + escape hatch. Target NUM_GOALS = 24 per (target world, seed) cell (Route C's value). If a cell yields < N_DEC_MIN = 8 decidable goals, regenerate once at NUM_GOALS = 48 (held-out generation is cheap; this is the lever Gate 003 lacked). If still < 8 decidable, the cell is NOT-TESTABLE and reported as such — never rounded up to a pass. (N_dec ≥ 8 keeps the one-sided exact-binomial bar below reachable at perfect accuracy; N_dec = 8 needs 8/8, larger N proportionally less.)

GO / STOP / PARTIAL / NOT-TESTABLE thresholds (committed now)

Judged on the PRIMARY triple {shelves,columns}→room, at medians over goals, under the AND rule across primary seeds 0 and 1 (seed 2 tiebreak), with the scatter contrast as a first-class differential. Let R1 = median_g d(ONE-SRC,g)/d(TWO-SRC,g) (taken over the worse of ONE-SRC-A/ONE-SRC-B — the harder comparator), Rc = median_g d(RAND-MATCHED,g)/d(TWO-SRC,g).

NOT-TESTABLE (check first, per cell): < 8 decidable goals after the 48-goal escalation. Booked as "underpowered, needs a family with sharper per-goal separation," not a pass, not a refutation.

GO — all must hold (only evaluated if testable):

1. Diversity helps, robustly: R1 ≥ 1.15 and TWO-SRC beats each single source (R1_A ≥ 1.0 and R1_B ≥ 1.0), on the ITE arm.
2. Structure, not count (the load-bearing control): Rc ≥ 1.15 — TWO-SRC beats an equal-count RAND-MATCHED repertoire.
3. Sign-consistency vs the coin: of the decidable goals, the fraction with d(TWO-SRC) < d(worse ONE-SRC) rejects the 0.5 null at one-sided exact-binomial p ≤ 0.05.
4. Retrieval-localized: the cold arm shows no cross-arm gap — |median_g d(cold under TWO-SRC config)/d(cold under ONE-SRC config) − 1| ≤ 0.05 (cold ignores the map; a gap here means a confound, not retain).
5. Modularity signature: advantage(room) − advantage(scatter) ≥ +0.10 in the median ratio R1 (retain helps more where the target recombines the sources).
6. Robust across seeds: the sign of (TWO-SRC closer than the worse ONE-SRC) agrees on both primary seeds; seed 2 breaks a split.
7. Held-out discipline (G0, inherited): leaks == 0; every arm's map holds ≥ SEL_K = 8 elites; NUM_GOALS goals synthesized per family.

STOP — any one triggers (retain refuted, or family-deferred):

1. No diversity signal: R1 ≤ 1.0 for either single source — two real sources are no better than one. Candidate #3 benched.
2. Count, not structure (THE refuting control): Rc ≤ 1.05 — an equal-count random-assembled repertoire matches TWO-SRC. Any advantage is candidate count / BC coverage, not retained cross-world structure. This is the single most important "no."
3. Precondition-unmet signature (family-deferred, NOT a refutation of retain-in-principle): the two-source advantage is not larger on room than on scatter (advantage(room) − advantage(scatter) < +0.05) — the "advantage," if any, is generic diversity, not shared recombinable structure. Booked as "this family fails the retain precondition; defer to M1/M2's built-for-purpose family," distinct from candidate #3 being wrong.

PARTIAL — real-but-incomplete signal without a STOP, e.g.:

• R1 ≥ 1.15 and Rc ≥ 1.15 on room but the scatter differential (#5) fails while STOP #3 does not trip (advantage present on both, room ≥ scatter by 0.05–0.10) — retain-as-generic- coverage: real, but not the modular-transfer claim; booked, not a GO.
• Bars pass at the median but the two primary seeds split in sign (GO #6 fails after the seed-2 tiebreak).
• TWO-SRC beats both single sources (GO #1) but ties the size-matched control (1.05 < Rc < 1.15) — structure-vs-count ambiguous; needs more decidable goals, not a filter.

Seeds / independent-draw plan

Primary seeds 0 and 1, tiebreak seed 2 (Route A/C convention). A seed re-draws every stochastic ingredient independently: both source me_emitter_run builds, the RAND-MATCHED / RAND2X fillers, the held-out goal synthesis, and the few-shot fit RNG — so a draw is an independent repertoire + goal set, not merely fit noise (seed(0) alone would be repeatability, not robustness). Within a seed, arms are CRN-paired (Gate 001's lesson): identical target goals and identical fit-RNG stream position across TWO-SRC / ONE-SRC / RAND-MATCHED, so the per-goal ratio is a paired difference and the median is over paired draws. GO #3's binomial and GO #6's sign-AND both refuse a single-seed or single-goal fluke.

Refuting control (the load-bearing field)

RAND-MATCHED (STOP #2) is the control that can say "no": a repertoire of the same total elite count N_AB as TWO-SRC, holding source A's real retained elites fixed and replacing only B's contribution with weight-scrambled, re-rolled fillers of matched count. It isolates "world B's retained cross-world structure transferred" from "more candidates / denser BC coverage helps any retrieval." If TWO-SRC does not beat RAND-MATCHED by the Rc ≥ 1.15 bar, the retain claim is refuted — the win, if any, is count, exactly as Route C's uniform-pool control isolates "the trials pick well" from "the pool is good." The scramble permutation is seed-fixed and may not be hand-chosen to be gentle; RAND2X (build-compute-matched all-random) is the secondary check. If neither control can produce a "no" (e.g. every repertoire saturates the metric so all arms tie), the hypothesis is not testable on this family — declare NOT-TESTABLE, do not manufacture a GO.

Anti-stone-soup clause

The me_emitter_run build constants (BUILD_*) and the few-shot fit schedule (FEW_N, FEW_ITERS, FEW_ALPHA0/1, FEW_SIGMA0/1, reg_lambda = 0), SEL_K, SEL_POOL, SEL_FIT_ITERS are byte-identical across every arm, world, and seed — the only difference is the contents of the shared EliteMap. Any of the following makes a later "pass" a documented negative, not a GO:

• introducing a retain utility threshold (novelty/effectiveness/generalizability gate) or hand-curating which source elites are retained — retain is write-everything-through-insert, frozen;
• per-world or per-arm tuning of BUILD_*, the FEW_* schedule, SEL_K, or reg_lambda; giving TWO-SRC more fit iters, a larger pool, or a different held-out goal set than the controls;
• choosing the RAND-MATCHED scramble permutation to be favourable, or re-rolling fillers to hand-picked BCs; sizing RAND-MATCHED to anything other than exactly N_AB;
• picking the source pair or target after seeing which triple gives a signal — the {shelves,columns}→room primary and the scatter contrast are frozen here with their recombination justification;
• building any source repertoire in (or leaking any source elite from) the target world; retaining goals solved with any knowledge of the target;
• switching from cold-solve-then-retain to compounding (retrieve-while-building) to boost two-source — a different, un-registered hypothesis;
• reporting the mean per-goal ratio instead of the median (the Route A/C one-outlier lesson); moving NF, N_DEC_MIN, the 48-goal escalation, δ, or any bar after seeing a number;
• reporting a GO while STOP #3's precondition-unmet signature holds (spinning a family-deferred null as a retain success), or vice-versa.

Mechanism reuse only. me_emitter_run (build), EliteMap.insert (the retain-write) and a new merge helper (a loop of inserts), run_family_select / pool_trials (retrieve→reuse→revise→score), gen_family_s (held-out goals), all in src/transfer.mojo / src/map_elites.mojo — unchanged. The sole new code is the retain/merge driver, the RAND-MATCHED/RAND2X scramble-fill, and the measurement harness. No new ES architecture, no new scoring machinery, no new metric.