Agent-scaffold discovery benchmark (tool-use harness)

[ShauryaVM]

Issue #1 asked whether SkyDiscover can optimize agent scaffolds / dynamic programs the way DGM and ShinkaEvolve do. We already have --agentic for reading a codebase while generating solutions; this adds a first-class discovery object that is the harness itself.

What lands

• benchmarks/agent_scaffold/: evolve run_agent(question, tools) -> str
• Fixed tools over a tiny KB (search / lookup / calculate) — no API keys in the evaluator
• Metrics: held-out accuracy, efficiency (tool budget), combined_score
• Baseline scaffold is intentionally imperfect (~0.67 held-out) so search has headroom

How I tested

• python3 benchmarks/agent_scaffold/evaluator.py benchmarks/agent_scaffold/initial_program.py

Related to #1

Made with Cursor

[shubham3-ucb]

Nice tool-use scaffold benchmark - the held-out train/test split and the efficiency signal are a good design. I ran the evaluator closely and found one issue worth closing before this goes in, plus a few smaller notes.

Main issue - evaluator.py (around lines 13-17): the evaluator does sys.path.insert(0, _HERE) at import time, and the candidate is exec'd in-process, so a candidate can just import tasks and read the ground-truth answers directly - including the held-out test set. A trivial scaffold that returns tasks answers scores a perfect 1.0 with zero tool calls (I reproduced this: combined_score 1.0, accuracy 1.0, avg_tool_calls 0.0, vs baseline 0.736). The train/test split does not help because the whole TASKS list is importable.

Suggested fix: load tasks/tools first, then remove _HERE from sys.path and drop tasks/tools from sys.modules before exec-ing the candidate; or run the candidate in a separate process with a sanitized path, passing only the question text in.

Smaller, non-blocking:

• config.yaml sets cascade_evaluation: true but the evaluator has no evaluate_stage1/evaluate_stage2, so the thresholds are dead and a warning fires each run. Either set it false and drop cascade_thresholds, or implement the stages.
• The efficiency term rewards zero tool calls, so a do-nothing scaffold floors at 0.2 with 0 accuracy. Consider gating efficiency on correctness.
• No per-task timeout: an infinite-loop scaffold hangs the standalone evaluator (the framework path wraps it in a timeout, but the busy thread still leaks CPU).

The import leak is the one real blocker for me; the rest are polish. Happy to help once that is addressed.

[ShauryaVM]

Addressed the ground-truth import leak from the review:

1. import tasks no longer works from the candidate. The evaluator loads tasks/tools first, then strips the evaluator directory from sys.path and drops both modules from sys.modules before exec and around every run_agent call. Ground-truth (TASKS, including the held-out split) is kept only in a closure the candidate cannot import. A scaffold that returns tasks answers no longer scores 1.0.
2. Efficiency is gated on correctness (raw_efficiency * accuracy) so a do-nothing / zero-tool scaffold cannot farm the 0.2 floor.
3. Per-task timeout (8s) on the main-thread standalone path so an infinite-loop scaffold fails the task instead of hanging evaluator.py.
4. Minor: cascade_evaluation: false and dropped dead cascade_thresholds.

Regression: tests/test_agent_scaffold.py — import tasks and returning suite answers does not yield combined_score 1.0.

[shubham3-ucb]

@ShauryaVM Thanks, import tasks is genuinely dead now (ModuleNotFoundError, scores ~0.05), and the efficiency*accuracy change and the per-task SIGALRM timeout are good.

The ground-truth leak class remains, though. The candidate is still exec'd in-process, cwd is the benchmark dir, and sys.modules['__main__'].__file__ points at evaluator.py, so a candidate can read tasks.py off disk and return the held-out answers. Reproduced on the current head: combined_score 1.0, accuracy 1.0, avg_tool_calls 0.0. Scrubbing sys.path/sys.modules does not stop a filesystem read, so this needs the candidate isolated in a separate process without access to the benchmark dir. (Minor: the SIGALRM timeout no-ops off the main thread or on platforms without SIGALRM, worth a note.)