Post-NAS MLP topology microbenchmark

[ShauryaVM]

Issue #2 asked about Post-NAS / architecture adaptation as a discovery domain. This is a small, dependency-light wedge: the evolved object is an MLP layer graph, not a one-shot algorithm.

What lands

• `benchmarks/post_nas_mlp/`: evolve `build_architecture(...)` (linear / activations / dropout)
• Pure-NumPy trainer (no torch) on a synthetic 3-class nonlinear dataset
• Metrics: `accuracy`, `n_params`, `combined_score` (accuracy + light size bonus)

How I tested

• `python3 benchmarks/post_nas_mlp/evaluator.py benchmarks/post_nas_mlp/initial_program.py`

Related to #2

Made with Cursor

[shubham3-ucb]

Clean, minimal post-NAS MLP microbenchmark - the accuracy-plus-complexity objective and the real validator are nicely done. I found one integrity issue worth closing before merge, plus one small note.

Main issue - evaluator.py (around lines 215-218): evaluate() calls the untrusted build_architecture before it resolves _train_eval, and _train_eval is looked up from the module's global namespace at call time. Because the framework runs the evaluator in a thread executor (same process), a candidate can walk sys.modules, find the running evaluator module, and rebind _train_eval to return perfect metrics. I reproduced this both standalone and through the real framework: combined_score 1.0 with a trivial net and no real search (honest baseline is 0.760).

Suggested fix: capture only the returned architecture (deep-copy it), and call _validate/_train_eval through local references captured before exec (or from a module the candidate cannot reach); or run the candidate in a subprocess / isolated namespace.

Smaller, non-blocking:

• config.yaml sets cascade_evaluation: true but there are no evaluate_stage1/evaluate_stage2 functions, so the thresholds are dead config and a warning fires each run. Set it false and drop cascade_thresholds, or implement the stages.

The scoring-isolation gap is the one real blocker for me; otherwise this is tidy. Glad to help once it is addressed.

[ShauryaVM]

Addressed the scoring-isolation issue from the review:

1. Trusted trainer is captured before candidate exec. evaluate is a closure over _validate / _train_eval / dims bound at module import. Candidates that walk sys.modules and rebind evaluator._train_eval no longer affect scoring (including later evaluations in the same process). The architecture returned by build_architecture is deepcopy'd before validate/train.
2. Trainer helpers are bound at definition time (make_dataset=... default args) so rebinding evaluator globals cannot redirect _train_eval's internals either.
3. Minor: cascade_evaluation: false and dropped dead cascade_thresholds.

Regression: tests/test_post_nas_mlp.py — a trivial net that rebinds _train_eval to perfect metrics no longer scores 1.0 (honest training path is used).

[shubham3-ucb]

@ShauryaVM Thanks, the module-global rebind is genuinely dead now: setting mod._train_eval scores the real result (~0.65), not 1.0.

The isolation is not real yet, though. The captured trainer lives in a writable CPython closure cell, and cell_contents is assignable. A candidate can walk sys.modules to evaluate, set evaluate.__closure__[i].cell_contents = fake_train_eval, and score combined_score 1.0 with a trivial 1-layer net. It also persistently poisons the process, so later honest evaluations return 1.0 too. Reproduced on the current head.

One more thing: the regression test only exercises the module-global rebind, so it passes while the closure variant still works. Closure capture is not isolation when cells are mutable and reachable; this needs a separate process.