Hypothesis→experiment scientific discovery benchmark

[ShauryaVM]

Wedge for scientific discovery beyond coding contests: the candidate designs which points to query on a hidden noisy oracle, fits a hypothesis, and is scored by noiseless held-out R².

What lands

• `benchmarks/hypothesis_experiment/` with `design_experiments` + `fit_hypothesis`
• Hidden target: `sin(x0) + 0.5 x1 x2 - 0.25 x0²`
• Baseline: random queries + degree-2 least squares

How I tested

• `python3 benchmarks/hypothesis_experiment/evaluator.py benchmarks/hypothesis_experiment/initial_program.py`

Made with Cursor

[shubham3-ucb]

Thanks @ShauryaVM, love the framing. Two blockers before merge, both leak the hidden target (both reproduced live):

1. (critical) evaluator.py:14-16 insert the evaluator dir onto sys.path before loading the candidate, so world.true_function is importable. A candidate whose fit_hypothesis returns lambda x: world.true_function(x) gets the noiseless truth. PoC: {combined_score: 1.0, r2: 1.0, mse: 0.0}. Fix: load world in an isolated namespace unreachable from the candidate, or sandbox candidate execution.

2. (high) evaluator.py:55 uses one rng = np.random.default_rng(0), passed to design_experiments, then reused for oracle noise and to draw the held-out x_test = sample_inputs(TEST_N, rng). A candidate can snapshot rng.bit_generator.state, skip the 40 oracle-noise draws, and reconstruct x_test exactly. PoC: reconstructed == actual x_test: True, max abs diff 0.0. Fix: use an independent test generator (e.g. rng.spawn()) that is never handed to candidate code.

Minor: world.py hardcodes default_rng(0) (world.SEED unused), so the test set is identical every run; consider rotating/undisclosed seeds and averaging. config.yaml:37 has the same dead cascade_thresholds.

Happy to re-review once these are in.

[ShauryaVM]

Addressed the isolation leaks from the review:

1. Hidden world is no longer importable. world.py is exec'd into a private namespace (never sys.modules['world']). During candidate exec / design_experiments / fit_hypothesis / predict, the evaluator directory is stripped from sys.path and any world module is dropped, so import world / world.true_function fails.
2. Independent test RNG. Candidate, oracle-noise, and held-out test generators are spawned from undisclosed entropy (Generator.spawn / SeedSequence.spawn). The test generator is never handed to candidate code, so snapshotting rng.bit_generator.state and skipping 40 noise draws can no longer reconstruct x_test.
3. Minor: removed unused world.SEED; set cascade_evaluation: false and dropped dead cascade_thresholds.

Regression: tests/test_hypothesis_experiment.py — a cheat that tries import world plus RNG replay no longer scores 1.0 (import path errors to 0.0; baseline still scores in (0, 1)).

[shubham3-ucb]

@ShauryaVM Thanks, the two named leaks are genuinely closed: import world now raises ModuleNotFoundError (scores 0.0), and the spawned independent RNG defeats the state-replay reconstruction of x_test.

The oracle-leak class is still open though. Candidate code runs in the same process and call stack as evaluate(), which holds true_function as a local. A predict() that walks sys._getframe().f_back reads it directly. Reproduced on the current head: combined_score 1.0, r2 1.0, mse 0.0. The sandbox blocks imports and sys.path, not frame access, so closing this needs real isolation (run the candidate in a separate process), not sys.path/sys.modules scrubbing.