Remove the reports (analyzer) feature - #1287
Open
kyle-compute wants to merge 1 commit into
Open
Conversation
PR A of the analysis-as-trials rebuild: delete the reports/evals stack end-to-end. Gone: the oddish evals/ package, reports core + CLI (core/analyzers.py, cli/report.py), the report worker arm (workers/queue/analyzer_handler.py — verified report-only), the backend sandbox-LLM stack (api/services/sandbox/, sandbox_llm_client.py, worker/analyzer_sandbox.py), report prompt/parse/runner blocks, ops entrypoints, the analyzers UI, and their test suites — 94 files, 13,369 lines. Migration dropanalyzers01 (chained on agentcap01) drops analyzers + analyzer_experiments. analyzer_blocks stays: summaries and capabilities still write it, and its analyzer_id is a plain string with no FK. ANALYZER worker jobs keep their trajectory_summary and agent_capabilities modes; any surviving report-mode job now fails terminally instead of retrying into a handler that no longer exists. The cleanup sweep's ANALYZER domain-mirror arm is removed with the table it mirrored into. LLMClientType.SANDBOX keeps its enum value for historical analyzer_blocks rows but has no registered backend; post_trial_sandbox_enabled (default false) must stay off until the classifier dies in the cutover PR.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Oddish previewCommit:
Vercel deployment URL: https://oddish-8cytup76z.oddish.app Plan:
This comment is updated by the PR Preview workflow. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR deletes the reports feature.
It is PR A of six (A–F) rebuilding analysis so every analysis job runs as a normal trial instead of through its own separate pipeline. This is the rebuild of [#1122](#1122).
This PR only deletes old behavior. It does not add the replacement yet.
What is being deleted
Reports let a user select completed eval runs and have an LLM generate a written analysis of them.
That feature had its own:
analyzersandanalyzer_experimentsANALYZERjobs withmode = "report"backend/api/routers/reports.pyoddish report/analyzerspagesThis PR removes all of it.
In total:
Examples:
AGENTS.mdno longer documents reportsDatabase change
This PR adds one migration:
It drops:
in that order because
analyzer_experimentsis the child table.No surviving table has a foreign key to either one, so nothing else is deleted.
The downgrade recreates both tables and their indexes.
What happens to already-queued report jobs
A worker may still receive an old job like:
after this deploy.
Instead of crashing or retrying forever, the dispatcher permanently fails it once with:
Permanent failure means the job is marked failed and is not retried.
What is intentionally kept
Three related things remain for later PRs:
analyzer_blocksstays because live QA data still reads it. PR E removes it.post_trial_sandbox_enabledstays until PR D.Until PR D merges, the Modal environment must not set:
The code behind that flag is deleted in this PR.
Verification
The test suites are already red on a fresh database on unmodified
staging, so the useful check is whether this branch introduces new failures.I ran this branch and
stagingagainst the same local Postgres 16 setup and compared the failure sets.Result:
oddish
backend
I also verified:
alembic upgrade headsucceeds from an empty Postgres databasedropanalyzers01analyzersandanalyzer_experimentsare goneanalyzer_blocksandtrialsremainimport oddishsucceedsimport backendsucceedspnpm buildexits 0/analyzersrouteTests
No new tests are added.
This is a deletion PR: tests whose only subject was deleted are deleted with it. Nothing is added solely to prove that removed code is absent.