Run every analysis job as a trial by pfbyjy · Pull Request #1122 · abundant-ai/oddish

pfbyjy · 2026-08-07T10:19:08Z

The platform runs agents to analyze its own results: QA and task audits. Each one had its own way to run. This PR deletes those paths and runs them all as normal trials. It also deletes the reports feature, because nobody used it.

Before and after

Before:

QA ran a Claude process directly on the worker machine.
Reports ran in a separate sandbox system left over from the chat feature.
Each path had its own retry, cost, and recovery code.

After:

All analysis runs as trials. A new column, trials.kind, marks them: agent, qa, audit.
An analysis trial is a regular trial with its own task image (python plus node). It does not run inside the audited task's environment. It reads the task and the trial logs through the oddish-query CLI.
Each analysis trial has its own verifier. The verifier checks that the agent wrote a valid JSON result with the required keys, stages it for collection, and rewards 1.0. A missing or invalid file fails the verifier, and the normal trial retries re-run the agent.
A finished analysis trial leaves one JSON file. An importer copies it into the same database columns as before. Dashboards, GitHub comments, and alerts do not change.
Analysis trials live in a hidden "qa report" experiment, one per experiment. The experiment page, the task page, and the dashboard link to it.
Trial retries replace all the special recovery code. Workers hold no LLM keys.
The reports (analyzer) feature is gone: the tables, the endpoints, the dashboard tab, and the evals package.

The life of one task now: the agent trials run. The last one finishes. One QA trial starts. It reads the logs, grades each trial, and writes the verdict. The verifier checks the file. The importer stores the result.

This branch includes the staging merge with #1120. The verdict now says accept or reject. When new trials are added to a task, the old verdict stays until the new QA run replaces it.

Numbers

223 files. +2,150 / −29,443.

Before you merge

Run the gold-label sweep. It grades the new QA against trials with known answers. If the score matches the old path, merge. This is why the PR is a draft. Note: the QA prompt text changed again — this PR embeds the prompts in the trial brief, Use structured outputs for trajectory summaries #1121 changed the summary prompt, and QA verdict: accept/reject labels, plain prompt, keep verdict until replaced, scoped rerun #1120 rewrote verdict_prompt.txt. The sweep must run on this branch's prompts.
QA and ANALYZER jobs in flight at deploy time fail. A migration cancels them, and the cleanup sweep restarts their work as trials. No task stays stuck.
One direct LLM call remains: the probe transcript summarizer. That is a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

vercel · 2026-08-07T10:19:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 10, 2026 10:35pm

github-actions · 2026-08-07T10:26:54Z

Oddish preview

Commit: 6a774595c255ea3d14a23e16309b89c7951e170c

Surface	Link	Target
Frontend	https://pr-1122.oddish.app	Vercel preview for `6a77459`
Backend	oddish-pr-1122	`oddish-pr-1122`
Database	project zwfvdtxcqyviivsesitg	`project zwfvdtxcqyviivsesitg`

Vercel deployment URL: https://oddish-d7f25rccz.oddish.app

Plan:

Frontend deploy: true
Backend deploy: true
Migrations: true

This comment is updated by the PR Preview workflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

trials.experiment_id is NOT NULL. Analysis trials borrow the task's first live experiment membership, falling back to any live trial's experiment. The analyzer host task gets a dedicated experiment so report trials never land in a user's. The experiment-page trial loader now filters kind='agent', since analysis trials carry experiment ids after this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Analysis trials need an experiment. Putting them in the experiment they grade polluted every experiment-scoped surface, so each experiment now gets one shadow: a hidden experiment named '<name> (qa report)' that holds the qa and audit trials for its tasks. The experiment page links both directions, the task page lists the shadow like any membership, and the QA page lists all reports. Analyzer report trials keep their own host experiment. QA results now carry _graded_by (the qa trial id) so the UI can jump from a graded trial to the trial that graded it, and the analysis trials get their own drawer group on the task page. Also: a Daytona teardown that finds the sandbox already gone is a success, not an error, and the post-trial hooks are guarded so a hook failure can never wedge a settled job -- the cleanup sweep re-runs stage advancement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The dashboard rows now carry the report id, so each experiment links to its qa report in place. The QA-page report listing goes away -- one way to get there, not two. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The qa cost views read the analysis_costs ledger, which the deleted block framework wrote. QA and audit now run as trials, so their spend sits on the trial row. Task scope gains a third union branch over analysis trials; experiment scope counts member tasks' analysis trials, owned when they live in this experiment's qa-report shadow. Legacy ledger rows still count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The qa agent did its work and wrote /logs/qa_result.json, but harbor only collects the agent/ and verifier/ subtrees -- a loose file at the /logs root never reaches storage. The import then found nothing. The analysis overlay now replaces the audited task's tests with a small verifier: it stages /logs/<artifact> under the collected verifier dir, checks it parses as JSON, and rewards 1.0. A missing or invalid file fails the verifier, so the normal trial retries re-run the agent. This also stops analysis trials from running the audited task's own verifier, which could burn LLM-judge spend on an agent that never tried to solve the task. The importer stops guessing fixed keys: it lists the trial prefix, prefers the staged verifier copy, and takes the newest match across retry attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

A file with just '{}' parsed fine, earned reward 1.0, and only failed later at import -- with no retry, because the trial had already succeeded. The verifier now also checks the keys the importer reads (trials + verdict for qa, items for audit), so a wrong-shaped file fails in the sandbox and the trial retries re-run the agent. No RewardKit: the verifier runs inside the audited task's container, which can lack pip, python, or network, so it installs nothing and uses whatever runtime is present. Full schema validation stays host-side in the importer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

An analysis trial is a regular trial, so it gets a regular task: its own task.toml, its own python+node image, and its own verifier. The overlay stops patching the audited task's directory and replaces it completely. Nothing of the audited task reaches the sandbox -- the agent fetches its source, logs, and trajectories through the oddish-query CLI, the same way the gold harness audits from artifacts. This removes the whole class of unknown-image failures: the audited task's container could lack python or node (the analyzer host image itself lacked node, so the CLI its briefs depend on could not run), and one fixed image means one cached build across every analysis trial. The trial still belongs to the audited task in the database, so the task page, the graded-by links, and the QA barrier are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Nobody uses it. The per-task pipeline is the product: the pre-trial audit, the QA trial, and the verdict. The cross-experiment reports page was a separate, older feature with its own machinery -- host tasks, map/reduce trial fan-out, its own gather and rollup code -- and all of it goes: the pipeline module, the CRUD, the eval prompts, the router, the pages, and the tables (dropped by migration). The worker-job enum members stay so historical rows keep deserializing. Analysis trials are qa and audit now, nothing else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

…l, eager-load perf Conflicts and how they were resolved: - queue.py append path: staging's clear_inflight_verdict wins. A successful verdict now stays on the task until the new QA pass replaces it. The in-flight QA trial for the old set still gets cancelled. - qa_handler.py and the old verdict tests: deleted in this branch. Their accept/reject change lives in analyze/models.py and verdict_sync.py, which the trial-world importer already uses. - summarize_trajectory.py and its tests: deleted in this branch. Staging's change there was an eager-load fix for a call path that no longer exists. - CHANGELOG: kept both sides, added the missing entry for the reports feature removal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The guard's schema-unit list pointed at core/analyzers.py, which the reports removal deleted, so the CI check crashed on a missing file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The verdict badge said "Running QA..." while the panel below still said "Trial QA has not run yet. Run QA...". The two must agree, so the panel now shows "QA is running." while the run is live, and marks stale results during a rerun. A live qa/audit trial now counts as QA in progress even when the status flags are stale, the cancel button reads "Cancel QA" when only an analysis trial runs, and that cancel goes to the QA cancel endpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Preparing the preview database rotates the branch DB password to a fresh random value on every run, but the Modal secret was only republished when the backend also deployed. A frontend-only push on a backend PR rotated the password, skipped the publish, and left the running backend with a dead connection string: every API request returned 500 until the next backend deploy. Publish the secret every time the password rotates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The password rotation happens on every prepare run, so the tests now assert the Modal secret publish always follows -- asserting its absence was asserting the bug the previous commit fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The eager pre-trial audit links the shadow experiment before the agent trials link the real one, so experiments[0] -- the primary-experiment pick -- returned the shadow: the CLI printed 'witty-signal (qa report)' as the submitted experiment and task rows chipped the shadow. Both primary-experiment pickers now prefer non-shadow experiments, and the task-row / browse / version chips exclude shadows entirely, matching every other experiment list. The detail page still reaches the report through the experiment's shadow_of linkage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The experiment page is the primary surface, so it now answers the QA question without a detour through task pages: every task row carries an Accepted / Rejected / QA-running / QA-failed chip (driven by the same live-trial-aware predicate as the task page, so stale flags cannot lie), and the summary bar grows a 'QA verdicts' tile with the accepted/rejected/running/failed rollup whenever any task in the grid ran QA. Non-QA experiments keep their five tiles. The task page's non-agent trial section drops the 'Analysis' header with raw kind strings ('qa', 'audit') for a 'QA' section labelled in words: 'QA run', 'Pre-trial audit'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The reports feature went out with the analyzer tables, but the CLI command survived and would 404 against the deleted POST /reports endpoint. Remove the command, its registration, and its tests. Also retire the intermediate-design leftovers in the docs: AGENTS.md no longer describes analyzer_map/analyzer_reduce trials or lists QA and ANALYZER as active worker-job kinds (drop_legacy_jobs_001 cancels them; only TRIAL, TASK_EXPAND and TAG_PROJECT remain active), and the trials.kind / WorkerJobKind comments in db/models.py match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

vercel Bot temporarily deployed to Preview August 7, 2026 10:19 Inactive

github-actions Bot deployed to Preview August 7, 2026 10:26 View deployment

pfbyjy changed the base branch from claude/remove-chat-0ld9o0 to staging August 7, 2026 10:41

vercel Bot temporarily deployed to Preview August 7, 2026 10:45 Inactive

claude added 8 commits August 7, 2026 10:50

Add trials.kind

24fb8f1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Run QA, audits, and analyzer reports as trials

d10cd23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Delete the block framework, sandbox machinery, and QA orphan subsystem

25a02e4

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Exclude non-agent trial kinds from user-facing surfaces

f750b3e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Render analysis trials by kind; patch surviving tests

3587be7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Targeted tests, doc updates, changelog for the trial cutover

ce60ef9

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Slim the analysis-trial modules

322b18d

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Tighten the changelog entry

1a11041

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

pfbyjy force-pushed the claude/everything-is-a-trial branch from 13407ec to 1a11041 Compare August 7, 2026 10:50

vercel Bot temporarily deployed to Preview August 7, 2026 10:52 Inactive

Rewrite the analysis-trial tests so each states its rule plainly

554e8f7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

vercel Bot temporarily deployed to Preview August 7, 2026 10:53 Inactive

github-actions Bot deployed to Preview August 7, 2026 11:06 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 11:20 Inactive

github-actions Bot deployed to Preview August 7, 2026 11:26 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 11:58 Inactive

vercel Bot temporarily deployed to Preview August 7, 2026 12:04 Inactive

github-actions Bot deployed to Preview August 7, 2026 12:12 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 16:55 Inactive

github-actions Bot deployed to Preview August 7, 2026 17:00 View deployment

github-actions Bot deployed to Preview August 7, 2026 19:59 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 20:57 Inactive

github-actions Bot deployed to Preview August 7, 2026 21:03 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 22:07 Inactive

vercel Bot temporarily deployed to Preview August 7, 2026 22:12 Inactive

vercel Bot temporarily deployed to Preview August 7, 2026 22:50 Inactive

vercel Bot temporarily deployed to Preview August 7, 2026 23:05 Inactive

github-actions Bot deployed to Preview August 7, 2026 23:11 View deployment

vercel Bot temporarily deployed to Preview August 7, 2026 23:12 Inactive

github-actions Bot deployed to Preview August 7, 2026 23:18 View deployment

vercel Bot temporarily deployed to Preview August 10, 2026 20:45 Inactive

github-actions Bot deployed to Preview August 10, 2026 20:48 View deployment

Merge origin/staging: promote-workflow sha pinning and e2e test de-race

bbdffc6

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

vercel Bot temporarily deployed to Preview August 10, 2026 20:51 Inactive

github-actions Bot deployed to Preview August 10, 2026 20:54 View deployment

vercel Bot temporarily deployed to Preview August 10, 2026 20:58 Inactive

github-actions Bot deployed to Preview August 10, 2026 21:04 View deployment

claude added 4 commits August 10, 2026 22:33

vercel Bot deployed to Preview August 10, 2026 22:35 View deployment

Navigation Menu

Uh oh!

Run every analysis job as a trial - #1122

Run every analysis job as a trial#1122
pfbyjy wants to merge 32 commits into
stagingfrom
claude/everything-is-a-trial

pfbyjy commented Aug 7, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

pfbyjy commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before and after

Numbers

Before you merge

Uh oh!

vercel Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Oddish preview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pfbyjy commented Aug 7, 2026 •

edited

Loading

vercel Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading