skydiscover/extras/evolve_analyzer

[oprince]

post-mortem diagnostic tool for evolutionary optimization runs directly into skydiscover as an optional sub-package, following the existing extras/ pattern (alongside monitor/ and external/).

• Adds skydiscover/extras/evolve_analyzer/ with 26 source files: ingestion, quantitative (11 analyzers), qualitative (6 LLM judges), vendored llm/ layer, coordinator, report_synthesizer, dashboard, historical_db, and bundled config/
• Adds tests/extras/evolve_analyzer/ with 303 passing tests
• Adds [evolve-analyzer] optional dependency group to pyproject.toml
• Adds run-evolve-analysis and dashboard-evolve-analysis CLI entry points
• Fixes config file path in coordinator.py (parents[2] → parent) to match the new bundled layout

Details and samples can be found here

Install with: pip install skydiscover[evolve-analyzer]

[gemini-code-assist]

Code Review

This pull request introduces a new evolve-analyzer package, providing a suite of tools for post-mortem analysis of evolutionary code optimization experiments. The changes include CLI entry points for running analyses and viewing reports, a dashboard for visualizing results, and a comprehensive set of quantitative and qualitative analyzers (including LLM-based judges). My review identified several areas for improvement, primarily regarding resource management, error handling, and adherence to Python best practices, such as avoiding library-level sys.exit() calls and ensuring proper resource cleanup in database connections.

[oprince]

Hi @lynnliu030 and @shubham3-ucb 👋

Would you have time to review this PR? It adds an optional evolve-analyzer sub-package under extras/, following the existing pattern alongside monitor/ and external/.

It's a self-contained post-mortem diagnostic tool for evolutionary optimization runs — quantitative analyzers, LLM-based qualitative judges, a Streamlit dashboard, and 303 passing tests. The main touchpoints with the existing codebase are pyproject.toml (new optional dependency group) and the CI workflow.

Happy to walk through anything or break the review into sections if that's easier given the size. No rush — whenever you get a chance works great.

Thanks!
Orit

[shubham3-ucb]

@oprince this is a self-contained, opt-in subsystem and the architecture is reasonable. It is isolated from core skydiscover and the 355 tests pass. I reviewed it for crash-safety and data integrity and ran each item below, so the repros are copy-pasteable. These affect the correctness of the output, so I think they are worth addressing before the merge.

Data correctness (these change reported results):

1. _extract_score treats a real 0.0 as missing. _extract_score({"combined_score": 0.0, "score": 5.0}) returns 5.0. A run that legitimately scores 0.0 gets the wrong score and delta. If 0.0 is a valid score, the x or y selection should be x if x is not None else y.
2. The same function drops a top-level score when a metrics dict is present: _extract_score({"metrics": {"foo": 1}, "score": 3.0}) returns None.
3. _normalise_status maps anything unknown to success: _normalise_status("weird") and _normalise_status(None) both return "success", so an unrecognized failure counts toward the success rate.
4. analyze_convergence: time_to_best_fraction uses the iteration value rather than the index, so it can leave [0, 1]. Records whose iteration starts at 100 give time_to_best_fraction = 20.8. The same function also raises KeyError on a record missing iteration (the change_points line subscripts ["iteration"] while the rest of the function uses .get).
5. LLM cache key has no model identity. The call site is cache_call(client.invoke, self.cache_dir, prompt), so the same prompt against two different models returns the first model's cached answer. The module docstring says it keys on "prompt + model identity."
6. analyze_exploration uses random.sample with no seed, so structural_diversity_index changes run to run on identical input.
7. The GEPA log parser truncates scientific-notation scores: child_score=1.5e-3 is captured as 1.5 (the score group is [-\d.]+, which excludes e).

Safety / crashes:

8. dashboard.py:1340 runs eval() on an evidence string. eval("__import__('os').getpid()") executes; eval("foo") raises an uncaught NameError (the except only catches ValueError, SyntaxError). The same file uses ast.literal_eval for the analogous sentinel case lower down, which is the safe form.
9. _stars(3.5) raises TypeError ("●" * float). A non-integer rating crashes the panel (_stars(3), _stars(None) are fine).
10. Crash-sample rendering hard-subscripts s["error"] / s["iteration"]; a sample missing either key raises KeyError.

Minor: historical_db interpolates metric_name and filter keys as SQL identifiers (values are parametrized). It is fine today since callers pass literals, but worth validating against the known column set. CI is red only on black (23 files), so black skydiscover/ clears it.

Scope note: I verified crash-safety and the data-path items above by running them. I did not independently validate the statistical formulas in the analyzers or run the full pipeline end to end, so this is not a sign-off on numerical correctness.

[oprince]

@lynnliu030 @shubham3-ucb, would you like to have a live review session?