Add bilevel autoreason: an outer meta-loop that adapts the inner loop's search procedure between runs

[thedavidmurray]

What this adds

A bilevel layer over autoreason. The existing A/B/AB → blind-Borda tournament is the inner loop and is driven unchanged; it optimizes the artifact. This PR adds an outer loop that optimizes the search procedure the inner loop runs under. After an inner run, the outer loop reads the tournament trace, identifies the single most-severe failure signature, applies one candidate mechanism (a parameter or prompt change), re-runs the inner loop, and keeps the change only if it improves autoreason's own metrics under a quality gate. Otherwise it reverts.

Concretely this is a greedy, one-mechanism-at-a-time, accept-or-revert hill-climb over a small fixed mechanism set, not a general optimizer. The accept/revert gate is the whole safety story: a mechanism ships only if the tournament trace says it earned its place.

The shape is standard bilevel optimization, and specifically the one in Bilevel Autoresearch: Meta-Autoresearching Itself (Qu & Lu, arXiv:2603.23420): same model at both levels, the outer loop reading the inner loop's traces and injecting search mechanisms. That paper reports a 5x val_bpb improvement over the inner loop alone on Karpathy's GPT pretraining benchmark. That is their result on their benchmark, not a claim about this implementation, which is untested empirically (see What's verified). I'm not claiming the bilevel idea as novel.

This PR deliberately narrows the paper's design in one important way, and it is worth stating up front because a reader who knows the paper will notice it. Their outer loop is generative: the same LLM reads code and traces and writes new Python mechanisms at runtime. This PR's outer loop is deterministic: a rule-based classifier over the trace, selecting from a fixed, human-readable mechanism library, with no LLM call at the diagnosis step. The trade is intentional. It is cheaper (diagnosis is free), fully auditable (every decision is a named rule firing on named trace fields), and a sound first rung. The generative outer loop is the obvious follow-on once the deterministic seam is in place.

What this PR actually delivers, stated without inflation:

1. a mapping from autoreason's specific trace signals (per-round Borda tallies, rounds-to-converge, judge agreement) to a small, named mechanism set — autoreason is a natural inner loop precisely because it already emits exactly these signals;
2. a conservative accept/revert controller with a repeated-vote quality gate;
3. a tested reference implementation wired to the existing v2 runners.

The mapping (item 1) is the novel part and also the unvalidated part. The controller and the wiring are ordinary engineering, and they are what carries the correctness guarantees here.

Files:

• experiments/v2/mechanisms.py — diagnose(trace) -> (bottleneck, evidence, mechanism), a pure, rule-based classifier (no LLM, no network) that matches trace signatures in a documented severity order, returns the top one, the trace fields that fired it (evidence, logged so a human can audit why a mechanism was chosen), and one candidate mechanism. Unit-tested via python mechanisms.py.
• experiments/v2/run_bilevel.py — the outer runner. Flags follow the v2 convention (--model, --tasks, --iterations, --max-mechanism-trials, --gate-votes, --mechanisms on/off, --out-dir); writes results_bilevel/ in the existing JSON layout.
• One backward-compatible change to run_overnight.py: run_autoreason_pass gains an optional config=None argument. When omitted it reproduces today's behavior exactly. This is the seam the outer loop drives through, and it replaces the alternative of importing internal helpers and reimplementing the pass loop (see How it composes).
• One ## Bilevel Autoreason section in the README (method summary, the signature→mechanism table, run commands, the cost note below, and the citation).

Bottleneck → mechanism library

Each mechanism is a concrete config delta, not a description. Numeric mechanisms change one exposed parameter; prompt mechanisms swap in a variant bundled in this PR. Each mechanism applies all its deltas together in one atomic bundle, so an accept/revert is attributable to that one bundled mechanism. It applies one mechanism per iteration. A kept mechanism changes the inner trace, so the next iteration diagnoses a different bottleneck and injects a different mechanism, up to --max-mechanism-trials; a reverted mechanism leaves the trace unchanged, so re-diagnosis returns the same mechanism and the search stops. Thresholds are heuristic defaults exposed as constants, meant to be tuned against your data; they are not asserted to be the right values.

Trace signature (default threshold)	Failure mode	Injected mechanism (one atomic bundle of config deltas)
A wins in ≤ N rounds; B/AB never place	premature convergence	escalate the critic to hard mode, rotate a fresh adversarial lens each round, raise B temperature by one fixed step
judges unanimous every round	judge herding	give each judge a distinct evaluation lens, add a required-dissent slot (one judge argues the minority case)
B, AB, and A judged near-identical	candidate collapse	add the prompt constraint forbidding B from paraphrasing A, raise B temperature by one fixed step
never converges within iteration cap	oscillation	require a Borda margin to unseat the incumbent, deterministic tie-break toward the incumbent, shorten the convergence streak
winner alternates A↔B across rounds	position bias	add judges (7 > 3) so the existing per-round ballot shuffle averages out, lean on the incumbent tie-break

Mechanisms are drawn from adjacent search-under-uncertainty literatures — bandits for judge allocation, design-of-experiments for diversity, annealing schedules for B temperature — and they can trade one failure for another. Raising B temperature can induce oscillation; a tie-break toward the incumbent can mask a genuine B/AB win. That interaction is the reason the accept/revert gate exists. The controller keeps a mechanism only if it survives re-measurement; the library is not claimed to be internally conflict-free.

Severity ordering. Overlapping signatures resolve by a documented default order. The non-convergence checks (position bias, oscillation) are evaluated first — a loop that never reaches a fixed point has no stable trace for the other diagnoses to read. Within the converged branch, judge herding corrupts the Borda tallies that the convergence-shape failures (premature convergence, candidate collapse) are read from, so it outranks those two. The order is a single constant and is meant to be reordered against real data.

Accept / revert rule (the core decision)

Two metrics conflict routinely, so precedence is explicit.

• Quality is a hard gate, and it is not a single vote. The gate runs --gate-votes blind pairwise votes (mechanism-on artifact vs mechanism-off, positions randomized per vote). A single vote compares two stochastic outputs and cannot separate run variance from mechanism effect, so the default is > 1 and the mechanism must not lose the majority. Any majority loss reverts immediately, regardless of convergence speed. More votes cost more judge calls and lower the per-gate error; this is the knob that trades compute for gate reliability.
• Rounds-to-converge is the objective, subject to the gate. Given the quality gate passes, keep the mechanism only if rounds-to-converge strictly drops. A tie reverts (prefer the simpler, unmodified loop).

Why this does not reward premature convergence. Fewer rounds is the objective and premature convergence is a named failure, so the two look in tension. They are reconciled by the precedence: a mechanism that converges faster by cutting the search short produces a worse artifact, which loses the quality gate, which reverts it. The rounds objective is only ever pursued on artifacts that already passed quality. Premature convergence lowers rounds and quality; the gate catches it on the quality axis.

Compounding and its bound. Kept mechanisms carry into the next diagnosis, so accepted changes stack — an improvement to the inner loop becomes the base config the next diagnosis reasons over, and a chain of accepts can drift the search. Two things bound this. The gate re-runs on every trial, not just the first, so each new base config is re-validated. And with per-gate false-accept probability p and T trials, expected false accepts are at most p·T; small T (the --max-mechanism-trials default is low) and larger --gate-votes (which lowers p) keep that product small. This is a bound on drift, not a proof of stability. A long chain is not claimed to stay stable, which is exactly why T is meant to stay small and why the gate is the only thing standing between "compounding improvement" and "compounding drift" — and it runs every step.

Cost

The outer loop is not free and eval compute is this repo's binding constraint, so the arithmetic is explicit. Diagnosis itself costs nothing (no LLM call). Each mechanism trial costs one full inner tournament plus --gate-votes judge calls. Worst case per task is --max-mechanism-trials × (one inner run + --gate-votes votes). --max-mechanism-trials 0 is an exact no-op equal to running the tournament unchanged.

On expected value: because ties and quality-losses both revert, the modal outcome of any single trial is a revert. That is by design, not a defect. The revert is the safety property, and you only pay inner-run cost for candidates actually tried (diagnosis is free, so tasks with no matching signature cost nothing extra). The honest framing is that this spends bounded, opt-in compute to find the occasional mechanism that survives, and most trials will not. Start at --max-mechanism-trials 1 --gate-votes 3 on a few tasks to measure per-trial cost before widening. If nothing survives the gate on a given task set, that is a clean negative result and the feature reduces to a no-op.

How it composes with the existing repo

• Inner loop driven through one sanctioned seam. run_bilevel.py calls run_autoreason_pass(config=...) directly. It does not reimplement or copy the tournament loop, and it does not reach into internal helpers to rebuild one. The only change to run_overnight.py is the optional config parameter, which defaults to current behavior. This is strictly less coupling than importing un-versioned internals and duplicating the pass structure, and it gives the inner loop a documented extension point instead of an implicit dependency on its internals.
• Automates the manual step in run_46_remedies.py. That runner already threads a hand-picked config (margin / scope / plateau) through its pass loop, with a human choosing the remedy. This PR automates that one decision: the outer loop reads the trace and picks the remedy, then adds the keep-only-if-it-helped check. It automates a single existing manual choice; it does not generalize that file.

What's verified, and what isn't

This is a method plus reference implementation, not a benchmarked result. It has not been run on your harness — I can't reproduce your eval budget — so there are no benchmark numbers in this PR or the README, and none are implied. What is unproven is specifically the empirical benefit of the mapping, not the correctness of the control flow.

Verified locally, with tests included in this PR:

• Classifier (python mechanisms.py): every signature routes to its intended candidate in the defined severity order; overlapping signatures resolve to the higher-severity one; an empty or ambiguous trace routes to no-op. These tests confirm the classifier does what the routing table says. They do not claim the underlying failure taxonomy is correct — they check the routing table I wrote, nothing more.
• Controller (test_bilevel_controller.py): a fixture-driven dry run stubs call_llm with canned tournament traces and asserts the accept/revert logic end to end (accept on strict improvement with a passing gate, revert on quality loss, revert on a tie, correct config threading across a compounding chain, --max-mechanism-trials honored). Zero network calls. This is the core contribution's behavioral test, and it is in the PR rather than promised.
• Wiring: python -m py_compile and python run_bilevel.py --help confirm imports resolve against run_overnight.py and the CLI is wired.

What is not verified: whether any mechanism actually survives the gate on a real task set, and whether the trace-field mapping matches your exact summary schema (see the open question below). The controller is correct; whether the mechanisms help is an empirical question this PR does not answer.

The first real experiment on your side is --mechanisms on vs --mechanisms off on tasks 1–5, reporting rounds-to-converge and the gate vote outcomes (both written to bilevel_summary.json). The falsifiable prediction is simple: either some mechanism survives the gate and lowers rounds without losing quality, or none does and the feature is a measured no-op.

Compatibility

Additive plus one backward-compatible signature change: two new files in experiments/v2/, one README section, and run_autoreason_pass(config=None), which reproduces current behavior when config is omitted. No existing prompt, result file, or caller is affected; every current call site keeps its exact behavior. diagnose() reads rounds_to_converge, per-round Borda tallies, and judge ballots — each derived from fields the tournament already writes to its summary output. The adapter that maps them is small and isolated, so if a field is named differently on the target harness it is a localized change and the classifier is untouched.

One decision I'd like your call on

The objective is rounds_to_converge, read from the summary JSON. If the harness exposes a convergence signal you consider more meaningful (for example a stability-of-winner measure rather than raw round count), name the field and the controller can optimize that instead. It is a one-line change to the objective accessor, and you have better information than I do about which signal best reflects real convergence quality on these tasks.

---

🤖 Generated with Claude Code