Add Claude Code and Codex as the solution generator

[akrentsel]

Summary

• Adds claude_code as a new search type that runs Claude Code CLI inside a Docker container as a single-agent baseline
• Claude Code autonomously iterates on the solution using the evaluator directly, then the framework scores the final result
• Supports both Python evaluators (simple container mode) and Docker evaluators (privileged DinD mode)

Usage

skydiscover-run initial.py evaluator.py --search claude_code -m claude-sonnet-4-6 -i 20

-i / --iterations maps to --max-turns for the Claude Code CLI.

Test plan

• All 31 existing tests pass
• Imports, config parsing, and registry registration verified
• End-to-end run on circle_packing benchmark with Docker + real API key completed successfully
• Test with a Docker-based evaluator (DinD mode)

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the Claude Code CLI as a new, autonomous single-agent baseline within the SkyDiscover framework. The primary goal is to enable the system to leverage Claude Code's iterative problem-solving capabilities directly, with the framework scoring the final output. This enhancement broadens the range of AI-driven search strategies available, offering a self-contained solution that manages its own iteration loop and supports different evaluator types through containerization.

Highlights

• New Claude Code Baseline: Introduced 'claude_code' as an optional single-agent baseline, allowing the Claude Code CLI to autonomously iterate on solutions within a Docker container.
• Flexible Evaluator Support: The Claude Code baseline supports both Python evaluators (simple container mode) and Docker evaluators (privileged Docker-in-Docker mode) for diverse testing environments.
• Configuration and Integration: Added new configuration options for Claude Code, including max_turns and docker_image, and integrated it into the existing search type and controller/database registration system.
• Docker Environment Setup: Provided a dedicated Dockerfile and entrypoint script to build and manage the Claude Code runner image, ensuring proper isolation and functionality for the CLI within a container.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• skydiscover/cli.py
  • Added 'claude_code' to the list of recognized search types.
• skydiscover/config.py
  • Introduced ClaudeCodeConfig dataclass to define configuration parameters for the Claude Code baseline, including max_turns and docker_image.
  • Registered ClaudeCodeConfig in the DATABASE_CONFIG_CLASSES mapping.
• skydiscover/search/claude_code/controller.py
  • Added ClaudeCodeController to manage the execution of the Claude Code CLI within a Docker container.
  • Implemented logic for building the Claude Code runner image, saving evaluator images for DinD mode, and writing evaluation scripts.
  • Developed methods to generate a TASK.md file for Claude Code's context and to execute the Claude CLI with appropriate Docker commands and turn limits.
  • Included error handling for unsupported LLM models and temporary workspace management.
• skydiscover/search/claude_code/database.py
  • Added ClaudeCodeDatabase, a minimal database implementation that stores the final program result, as Claude Code handles its own internal iteration.
• skydiscover/search/claude_code/runner_image/Dockerfile
  • Created a Dockerfile to build the Claude Code runner image, installing necessary dependencies like Python, Node.js, Docker, and the @anthropic-ai/claude-code CLI.
  • Configured a non-root user (claude) and added it to the docker group for privileged operations.
• skydiscover/search/claude_code/runner_image/entrypoint.sh
  • Added an entrypoint script for the Claude Code Docker container, handling Docker-in-Docker (DinD) mode initialization, including starting the Docker daemon and loading evaluator images.
  • Implemented logic to ensure workspace writability and to drop privileges to the claude user before executing the main command.
• skydiscover/search/route.py
  • Imported ClaudeCodeController and ClaudeCodeDatabase.
  • Registered the new 'claude_code' search type with its corresponding controller and database.

Activity

• All 31 existing tests passed.
• Imports, config parsing, and registry registration were verified.
• An end-to-end run on the circle_packing benchmark with Docker and a real API key was completed successfully.
• Testing with a Docker-based evaluator (DinD mode) is pending.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new search type, claude_code, which integrates the Claude Code CLI as a single-agent baseline. The implementation is well-structured, featuring a new controller, a minimal database, and a dedicated Docker image to run the agent in an isolated environment. The handling of Docker-in-Docker for containerized evaluators and privilege management is thoughtful. I have a few suggestions to enhance robustness and align with best practices, including making the turn-counting logic more resilient, optimizing the Dockerfile, and adding an explicit check for the required API key.

[shubham3-ucb]

Issues found from testing

Ran this against a real benchmark. Analyzed the stream-json log (~2000 lines, 48 result events, single session). Here are the concrete issues:

1. --max-turns is per-segment, not per-run (critical)

Claude Code resets the turn budget after context compaction. In the test run, --max-turns 101 produced 5 segments × 101 = 505 actual turns, ~8 hours, ~$80. The CLI's own error_max_turns fires per-segment, then Claude compacts and starts a fresh segment with a new 101-turn budget.

The manual turn counter (L263) tries to catch this by counting "type":"user" events, but this only partially works — it mixes in subagent events (43 subagent sessions emitted "type":"user" too), inflating the count with noise.

Fix: Parse "type":"result" events from the stream-json and sum their num_turns field across all segments. Kill when the cumulative total hits the budget. This is the authoritative turn count — Claude Code itself reports it.

2. No wall-clock timeout

If a single turn hangs (API timeout, tool hang, evaluator stuck), nothing stops the process. Need a hard wall-clock limit as a safety net (e.g., max_turns * 120s).

3. checkpoint_callback is received but never called (L142, L291)

The framework passes checkpoint_callback into run_discovery, but it's never invoked. This means:

• No intermediate checkpoints saved to disk
• If the process dies after hours of work, all progress is lost
• The framework's checkpoint directory stays empty

Fix: Poll the solution file periodically (it's on a bind mount, readable from the host). When it changes, evaluate it via the framework's evaluator, add to database, and call checkpoint_callback.

4. No container name → no reliable cleanup

docker run --rm without --name means if the process hangs and we need to kill it externally, we have to find the container by image name. Add --name and docker rm -f in the finally block.

5. solution_path.read_text() can crash (L276)

If Claude Code fails early or never writes to the file, and initial_code was empty, this crashes with an unhandled exception. Should handle OSError and fall back to initial code.

Minor

• L265: logger.info on every single turn is noisy at 100 turns. Log every 5-10 or on segment boundaries.
• Container not cleaned up in finally block if process was killed.

[shubham3-ucb]

All review feedback has been addressed in commit 1fb93af on this branch. Summary of each fix:

@shubham3-ucb

1. --max-turns is per-segment, not per-run — Fixed. The controller now parses result events from the stream-json and accumulates num_turns across all segments. The process is killed as soon as the cumulative total reaches the budget. This is the authoritative count from the CLI itself, so context compaction no longer inflates the turn count.

2. No wall-clock timeout — Fixed. Added wall_timeout = max(max_turns * (120 + eval_timeout), 600). This gives each turn a full eval timeout plus 2 min of thinking time, so the safety net never fires before the turn budget is exhausted in normal operation.

3. checkpoint_callback never called — Fixed. A 60-second poller runs concurrently in the async event loop. When the solution file changes, it evaluates via the framework evaluator, adds to the database, and calls checkpoint_callback. Progress is now saved continuously, not just at the end.

4. No container name → no reliable cleanup — Fixed. The container is named skydiscover-cc-<uuid> and docker rm -f is called in the finally block, so cleanup happens on any exit path including exceptions and timeouts.

5. solution_path.read_text() crash — Fixed. Wrapped in try/except OSError with fallback to initial_code. Also added a guard for an empty file.

@gemini-code-assist

1. ANTHROPIC_API_KEY fails silently — Fixed. Now raises ValueError immediately with a clear message if the variable is unset.

2. Turn counting via string match is brittle — Fixed. The original substring approach is gone entirely. Turn counting now uses json.loads on each line and reads result.num_turns, which is both correct and robust to formatting changes.

3. Dockerfile: separate RUN layers, missing ca-certificates — Fixed. The two apt-get RUN instructions are merged into one, ca-certificates is added, and both apt installs use --no-install-recommends.

Additional fixes not in the original review

• run_eval.sh for Python evaluators now calls evaluate() directly and prints JSON to stdout. The original script ran python3 evaluator.py which exits silently if there is no __main__ block, giving Claude no feedback and no way to improve.
• TASK.md is written to .prompt.txt and fed to the CLI via stdin to avoid shell quoting issues with backticks and special characters in the task description.
• eval_timeout is now defined once before the is_docker_eval branch rather than duplicated in each arm.
• progress.log is written to the output directory with timestamped entries (tool calls, segment completions with accurate turn counts, checkpoint scores). Writes are protected by a threading.Lock since the file is written from both the executor thread and the async checkpoint loop.

[gemini-code-assist]

Thank you for the comprehensive update, @shubham3-ucb! It's great to see that the issues identified during testing, as well as the points I raised, have been thoroughly addressed in commit 1fb93af.

I particularly appreciate the detailed explanations for each fix:

• ANTHROPIC_API_KEY fails silently: The change to raise a ValueError immediately for an unset API key significantly improves error handling and developer experience. This is a high severity improvement for usability.
• Turn counting via string match is brittle: Switching to json.loads and reading result.num_turns is a robust and accurate solution for turn counting, addressing the brittleness of the previous approach. This is a high severity improvement for correctness.
• Dockerfile: separate RUN layers, missing ca-certificates: Merging apt-get commands, adding ca-certificates, and using --no-install-recommends are excellent practices for Docker image optimization and security. This is a medium severity improvement for maintainability and efficiency.

Beyond my specific feedback, the fixes for the critical issues you found during testing, such as the --max-turns logic, wall-clock timeout, checkpoint_callback invocation, container cleanup, and solution_path.read_text() crash, are crucial for the stability and reliability of the Claude Code integration. These demonstrate a very thorough and effective debugging process.

Overall, these changes significantly enhance the robustness, correctness, and maintainability of the new Claude Code baseline. Well done!

[shubham3-ucb]

Additional fix pushed in commit 2b85c9c:

Docker evaluator arg order bug — run_eval.sh was calling evaluate.sh "$MODE" "$CANDIDATE" but all 23 evaluate.sh scripts across the repo expect $1=program_path, $2=mode. This caused spec_from_file_location('train', ...) to receive the mode string as the file path → returned None → crash on every Docker evaluator call. Fixed by swapping to evaluate.sh "$CANDIDATE" "$MODE".

Verified: all 23 evaluate.sh files use the same consistent format ($1=program, $2=mode). Both Python and Docker eval paths are now confirmed correct across all benchmarks.

[mr-sarthakgupta]

Hi @shubham3-ucb I looked at branches add-cc and branch add-claude-code-clean where claude-code is being used for this repository. Please let me know if organized results from using claude-code instead of the simplistic agent could be made available anytime soonn

[shubham3-ucb]

@akrentsel Two things before this can land:

1. controller.py:94 emits evaluate.sh "$MODE" "$CANDIDATE", but the contract is evaluate.sh <PROGRAM> <MODE> (PROGRAM=$1, MODE=$2) used everywhere else. As written, every containerized eval tries to load a file named train and fails, so in-loop scoring is always zero. Swap to evaluate.sh "$CANDIDATE" "$MODE".

2. The docstring says isolated DinD with no host socket, but the run path bind-mounts the host /var/run/docker.sock and chmod 666s it into a claude --dangerously-skip-permissions container, which is root-equivalent host access. Either drop the socket mount (use the DinD path, currently dead code) or update the docstring to match reality.

Also needs black and a rebase onto main's existing claude_code impl.