Optimize-the-optimizer: evolve search controllers

[ShauryaVM]

Small EvoX² wedge: the candidate is a search controller (selection + mutation / `ask`), not a solution to one math task. The evaluator runs it on a black-box portfolio under a fixed budget and weights held-out Ackley / Rosenbrock most heavily.

What lands

• `benchmarks/optimize_the_optimizer/` with `SearchController` API
• Train suite: sphere / rastrigin; test suite: ackley / rosenbrock
• Pure NumPy, no LLM inside the evaluator

How I tested

• `python3 benchmarks/optimize_the_optimizer/evaluator.py benchmarks/optimize_the_optimizer/initial_program.py`

Made with Cursor

[shubham3-ucb]

Thanks @ShauryaVM, strong benchmark idea. Two blockers before merge, both harness integrity (both reproduced live):

1. evaluator.py:36 — run_fn = getattr(module, "run_controller", run_controller). Every candidate is a copy of initial_program.py, which defines run_controller outside the EVOLVE-BLOCK (line 70; block ends line 67), so the candidate always supplies its own scoring harness. A candidate that returns the per-problem ceiling (0.0) scores 1.0 with no search. PoC: a program whose run_controller returns 0.0 gives {combined_score: 1.0, test_score: 1.0, train_score: 1.0} in 0.001s. Fix: keep run_controller in a non-evolvable module (or evaluator.py) and never getattr it off the candidate; the candidate should only supply SearchController.

2. initial_program.py:94 — while remaining > 0 decrements only inside for x in proposals, so a controller whose ask() returns [] loops forever. It is a hang not an exception, so evaluate() try/except does not catch it and the framework run_in_executor future cannot kill the thread. PoC: a candidate whose ask() returns [] times out the evaluator (exit 124). Fix: if not proposals: break (or inject a random point).

Minor: config.yaml:40 sets cascade_evaluation: true with no evaluate_stage1/stage2, so cascade_thresholds are dead config and a warning fires every run. Set false or implement the stages.

Happy to re-review once these are in.

[ShauryaVM]

Addressed the harness-integrity issues from the review:

1. run_controller is no longer taken from the candidate. The evaluation loop now lives in evaluator.py and is always used. getattr(module, \"run_controller\", ...) is gone. A candidate whose run_controller returns 0.0 (the per-problem maximize ceiling) is ignored; only SearchController is loaded. Removed the unused harness from initial_program.py so the seed no longer copies a scoring loop.
2. Empty ask() no longer hangs. The budget loop does if not proposals: break, so a controller that returns [] exits instead of spinning forever (which try/except cannot catch).
3. Minor: cascade_evaluation: false and dropped dead cascade_thresholds.

Regression: tests/test_optimize_the_optimizer.py — stub run_controller does not score 1.0; empty ask() returns in <10s rather than hanging.

[shubham3-ucb]

@ShauryaVM Thanks, both blockers are genuinely fixed and I confirmed it: a candidate-supplied run_controller returning 0.0 no longer scores 1.0 (it runs your real controller), and an empty ask() now exits in ~0.01s instead of hanging.

One separate benchmark-design issue: every objective has its optimum at the origin (sphere, rastrigin, ackley all minimize at 0, and rosenbrock at the origin still scores ~0.995), so a do-nothing controller whose initial_population() returns [zeros] and ask() returns [] scores combined_score 0.998 with no search, beating the honest baseline (0.959). Worth shifting the optima off-origin (e.g. random per-problem offsets) so the benchmark rewards real search.