Run every analysis job as a trial

.github/scripts/preview/prepare_preview_database.sh

@@ -86,7 +86,7 @@ if [ "$schema_rebuilt" != "true" ] && { [ "$RUN_MIGRATIONS" = "true" ] || [ "$br
   ( cd "$GITHUB_WORKSPACE/backend" && uv run python "$script_dir/seed_preview_db.py" )
 fi
 
-if [ "$DEPLOY_BACKEND" = "true" ] || [ "$branch_was_created" = "true" ]; then
-  "$script_dir/publish_modal_db_secret.sh"
-  published_modal_secret=true
-fi
+# Unconditional: the supabase step rotates the DB password every run, so the
+# secret must carry the new value even when no backend deploy follows.
+"$script_dir/publish_modal_db_secret.sh"
+published_modal_secret=true

AGENTS.md

@@ -29,7 +29,6 @@ Python `3.13` is required for `oddish` and `backend`. Node.js `20+` and `pnpm` a
 oddish/                         # Core Python package (CLI, server, workers, DB)
 ├── src/oddish/
 │   ├── analyze/                # QA prompts and analysis helpers
-│   ├── blocks/                 # Block/AnalyzerBlock primitive + LLM backends
 │   ├── cli/                    # oddish run/upload/ls/status/cancel/pull/collect/...
 │   ├── core/                   # shared endpoint/service logic (reused by backend/)
 │   ├── server/                 # standalone FastAPI app (python -m oddish.server)
@@ -53,7 +52,7 @@ backend/                        # Hosted cloud layer (Modal deployment)
 ├── api/
 │   ├── app.py                  # FastAPI app factory and lifespan wiring
 │   ├── schemas.py              # Pydantic models for org/auth/share responses
-│   ├── services/               # hosted services (sandbox runtime, analyzer blocks, …)
+│   ├── services/               # hosted services (Slack unfurls, helpers)
 │   └── routers/                # tasks, trials, dashboard, documents, tags, skills,
 │                               # admin, orgs, api_keys, imports, load, webhooks
 ├── auth/                       # header parsing (auth/__init__.py), API key + Clerk JWT
@@ -101,7 +100,7 @@ FastAPI server — oddish standalone (python -m oddish.server)
         |
         v
 Postgres
-  - worker_jobs       # unified queue (TRIAL / QA / TASK_EXPAND / TAG_PROJECT / …)
+  - worker_jobs       # unified queue (TRIAL / TASK_EXPAND / TAG_PROJECT / …)
   - trials / tasks    # domain state + live UI columns
   - trial_events      # short-lived live transcript pages for running trials
   - queue_slots       # per-queue-key concurrency leases
@@ -128,11 +127,21 @@ High-level flow:
    failed attempts in normal UI/API trial sets.
 3. Workers claim one `worker_jobs` row at a time, dispatch to the registered
    handler for its kind, write heartbeats, and exit.
-4. Trajectory analysis is **task-scoped**: when every trial of a
-   `run_analysis` task is terminal, a single `QA` job is enqueued. That one
-   job classifies every live trial's trajectory (written to `trials.analysis`)
-   and then synthesizes the task verdict (`tasks.verdict`). A sweep of `T`
-   tasks × `N` trials therefore enqueues `T` QA jobs, not `T × (N + 1)`.
+4. Trajectory analysis is **task-scoped** and runs as a trial: when every
+   agent trial of a task is terminal, one QA trial (`trials.kind = 'qa'`)
+   is created on the same task. Its agent classifies
+   every live trial, writes per-trial trajectory summaries, and synthesizes
+   the task verdict into one artifact (`qa_result.json`); on settlement an
+   importer writes `trials.analysis`, `trials.trajectory_summary`, and
+   `tasks.verdict`. The verdict is only requested above an evidence bar
+   (≥5 QA-eligible trials from ≥3 agents, `MIN_VERDICT_TRIALS` /
+   `MIN_VERDICT_AGENTS` in `oddish.workers.analysis_trials`); below it the
+   QA trial still classifies trials and the task completes without a
+   verdict. A sweep of `T` tasks × `N` trials therefore creates `T`
+   QA trials, not `T × (N + 1)`. The pre-trial audit is an `audit`-kind trial
+   created once per task version at sweep time.
+   Non-'agent' kinds are excluded from cost, quota, leaderboard, facet, and
+   public surfaces (see `oddish.filters.EligibleTrialScope`).
 5. While a trial runs, a worker-side tailer (`oddish.workers.harbor.live_tail`,
    on by default via `live_tail_enabled` / `live_tail_interval_sec`) polls the
    agent's log file inside the sandbox for supported agents (claude-code,
@@ -199,45 +208,40 @@ files under `oddish/src/oddish/analyze/`: `prompts/pre_trial_qa.txt` drives the
 source audit, `classify_prompt.txt` drives the per-trial log classifier,
 `verdict_prompt.txt` drives verdict synthesis, and
 `prompts/trajectory_summary.txt` drives schema-v5 trajectory summaries; the
-summary template must retain the `{{taxonomy}}` placeholder rendered by the
-block. Editing a prompt is a code change that ships with a deploy.
+summary template must retain the `{{taxonomy}}` placeholder, rendered by the
+QA-trial brief builder (`oddish.workers.analysis_trials`). Editing a prompt is
+a code change that ships with a deploy.
 
 ### Worker job kinds
 
 `WorkerJobKind` (in `oddish.db.models`):
 
-- **Active**: `TRIAL` (Harbor trial execution), `QA` (task-level classify-all-trials +
-  verdict), `ANALYZER` (cross-experiment report orchestration),
-  `TASK_EXPAND` (sweep expansion), `TAG_PROJECT` (tag recompute).
-- **Legacy, drain-only**: `ANALYSIS` (per-trial classification; `AnalysisJobHandler`
-  is kept only so in-flight rows survive a deploy), `VERDICT` (enum value only,
-  no handler), and `ANALYZER_BLOCK` (executed rows of the removed
-  `analyzer_runs` table; enum value only, no handler). Nothing enqueues any
-  of them anymore.
-- **Reserved**: `QA_REVIEW` (enum value, no handler yet).
+- **Active**: `TRIAL` (Harbor trial execution — including `qa` and `audit`
+  kind trials), `TASK_EXPAND` (sweep expansion), `TAG_PROJECT` (tag
+  recompute).
+- **Legacy, enum-only**: `QA`, `VERDICT`, `ANALYSIS`, `QA_REVIEW`,
+  `ANALYZER`, `ANALYZER_BLOCK`. QA/audit/analyzer work runs as trials now;
+  nothing enqueues or handles these kinds, and `drop_legacy_jobs_001`
+  cancelled any still-queued rows. The members stay so the native
+  `worker_job_kind` Postgres type keeps the values historical rows
+  reference.
 
 ## Package Boundaries
 
 `oddish` owns the execution core and shared queue/runtime primitives:
 
 - core models and migrations, including `worker_jobs` and `queue_slots`
 - unified claim/dispatch SQL, one `run_single_worker_job` runner, and a
-  handler registry (`TrialJobHandler`, `QaJobHandler`, `TaskExpandJobHandler`,
-  `TagProjectJobHandler`, plus the legacy `AnalysisJobHandler`)
-- the task-level QA job (`run_task_qa_job`): classify every live trial via
-  the shared `classify_trial_and_store`, then synthesize the task verdict
-- the verdict state machine (`oddish.core.verdict_state`), which is the only
-  writer for `tasks.verdict*` lifecycle columns and preserves the last
-  published result until a replacement succeeds or terminally fails
-- post-trial classification runs through `AnalyzerBlock`. It reads two
-  already-downloaded directories and executes nothing, so `resolve_substrate`
-  keeps it on the worker-local Claude Code client (`CLAUDE_CLI`) everywhere;
-  `post_trial_sandbox_enabled` is the operator opt-in that lifts it into a
-  Daytona `SANDBOX`, which restores the task/trial snapshot at the worker's own
-  absolute paths so no prompt rewriting is involved. Its costs use the
-  `post_trial` job kind; the legacy `trial_classifier` cost bucket is retired at
-  this cutover, and every block row carries `block_metadata.cost_status`
-  (`recorded` | `no_usage` | `failed`) so lost spend is queryable.
+  handler registry (`TrialJobHandler`, `TaskExpandJobHandler`,
+  `TagProjectJobHandler`)
+- analysis trials (`oddish.workers.analysis_trials`): brief builders,
+  settlement importers, and the audit/QA pipeline edges. Workers execute no
+  LLM calls of their own (the one exception is the probe transcript
+  summarizer in `oddish/worker/probe_analysis.py`); every analysis agent
+  runs as a trial on the analysis model's queue key
+- the verdict state machine (`oddish.core.verdict_state`), the only writer
+  for `tasks.verdict*` lifecycle columns, which preserves the last published
+  result until a replacement QA pass succeeds or terminally fails
 - shared queue-slot leasing, per-queue-key concurrency limits, and
   per-user fairness on `TRIAL` claims
 - database-backed admin concurrency overrides; these take precedence over
@@ -252,26 +256,10 @@ block. Editing a prompt is a code change that ships with a deploy.
 - soft-delete semantics on domain rows via the `deleted_at` column and
   a session-level filter (`oddish.db.soft_delete`)
 
-`oddish/src/oddish/blocks/` holds the analyzer-block primitive (prompt
-building, streaming, `analyzer_blocks` + S3 persistence) and its API/OpenAI
-backends, so verdict synthesis runs in a backend-free worker. The Daytona
-sandbox backend needs the hosted sandbox runtime
-(`backend/api/services/sandbox/` — Daytona client, provisioner, Claude Code
-runtime) and stays in
-`backend/api/services/blocks/analyzer/sandbox_llm_client.py`, which registers
-itself into core's client factory on import. `AnalyzerBlock` owns a
-self-provisioned client's complete lifecycle, including sandbox file downloads
-before close. Hosted callers request sandbox capabilities declaratively; the
-registered Daytona factory owns runtime/CLI installation, short-lived internal
-key minting, and key/sandbox cleanup. Callers must not provision and inject a
-one-off sandbox client for those capabilities.
-
-Hosted failure analysis uses
-`backend/api/services/blocks/analyzer/analyzer_block_runner.py`: it partitions a bucket
-into map batches, runs independent sandbox-backed `AnalyzerBlock`s concurrently
-up to `AnalyzerEvalConfig.map_concurrency`, collects their findings artifacts
-host-side, and supplies those artifacts declaratively to a separate reduce
-block. Map/reduce blocks never share or receive a live runtime/client.
+Cross-experiment failure reports (the old Reports page, `POST /reports`, and
+the analyzer map/reduce pipeline) were removed entirely; `drop_analyzers_001`
+drops their tables. Trial-level trajectory analysis and the task verdict are
+the QA trial's job (above) — there is no separate report machinery.
 
 `oddish` must not import from `backend/`, `backend.auth`, `backend.models`,
 `cloud_policy`, `idempotency_store`, Clerk, or Modal app/deployment modules.
@@ -454,8 +442,6 @@ Behavior:
 | `worker_job_dispatcher.py` | `discover_active_worker_job_queue_keys`, `get_worker_job_org_queue_counts`, `build_spawn_plan` (org-first fair-share, with within-org round-robin across queue_keys) |
 | `worker_job_single_job.py` | `_CLAIM_WORKER_JOB_SQL`, `run_single_worker_job`, `heartbeat_worker_job` |
 | `trial_handler.py` | TRIAL execution body |
-| `qa_handler.py` | Task-level QA job: `run_task_qa_job` classifies every live trial then synthesizes the verdict |
-| `analysis_handler.py` | `classify_trial_and_store` (shared per-trial classifier) + the transitional `run_analysis_job` wrapper for in-flight legacy ANALYSIS rows |
 | `task_expand_handler.py` / `tag_project_handler.py` | TASK_EXPAND and TAG_PROJECT job bodies |
 | `cleanup.py` | Zombie reaper, stale-heartbeat sweep, stage safety nets, **per-slot** orphaned-slot release (see invariants below) |
 | `slots.py` | `queue_slots` lease acquire/release (`locked_by` / `locked_until` / `locked_at`) |
@@ -743,20 +729,17 @@ pipeline counts off the analysis/verdict *model*'s queue key: that folds
 pipeline state into a real model's bucket — an incident rendered 4k+ trials
 mid-classification as "running workers" under one model's queue while that
 model's actual trials were routed into the "analyses" pipeline. These are
-presentation buckets only; the task-level QA worker job still enqueues under
-`get_qa_queue_key()` (the analysis model's concurrency bucket).
-
-Related invariant: a QA job that dies or is cancelled mid-classification must
-not strand its trials in a non-terminal `analysis_status`. The stale-heartbeat
-QA mirror resets them (RETRYING → `QUEUED`, FAILED → `FAILED` with the
-`ORPHANED_ANALYSIS_ERROR_PREFIX` sentinel), the append-supersede cancel path
-requeues in-flight rows via `requeue_inflight_trial_analysis` (which also
-reopens sentinel-FAILED rows when an append resurrects the task), and
-`_reset_orphaned_trial_analysis` in the cleanup sweep is the backstop. If you
-add a new way to kill or cancel a QA job, reset its task's in-flight
-`analysis_status` the same way — and select the trial rows `FOR UPDATE SKIP
-LOCKED`: these writers may hold the task row lock, and *waiting* on trial rows
-inverts the trials-then-task lock order `cancel_tasks_runs` takes (deadlock).
+presentation buckets only; QA/audit/analyzer trials queue under
+`get_qa_queue_key()` (the analysis model's concurrency bucket) and are
+excluded from the per-queue trial scans by `kind = 'agent'`.
+
+Related invariant: a QA trial that dies retries like any trial; a terminal QA
+trial whose import never landed is re-imported by the VERDICT_PENDING healer
+in the cleanup sweep, which also creates a fresh QA trial when none exists.
+Appending trials to a task cancels its in-flight QA trial (stamped with the
+cancelled harbor_stage) so a stale import can't overwrite the new set's
+verdict; the importer additionally refuses to store a verdict while any live
+agent trial is non-terminal.
 
 ---
 

CHANGELOG.md

@@ -6,10 +6,90 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ---
 
+## [2026-08-12]
+
+### Changed
+
+- Analysis no longer has opt-ins. Every task runs QA when its agent trials
+  settle and every task version gets a pre-trial audit at sweep time; the
+  `--run-analysis` / `--enable-analysis` CLI flags, the `run_analysis` /
+  `enable_analysis` API fields, the per-org pre-trial setting (endpoints,
+  worker gate, dashboard route), and the `ODDISH_PRE_TRIAL_ENABLED` env flag
+  are all gone. The `tasks.run_analysis` column stays (always true) for old
+  readers.
+- The task verdict now requires evidence: at least 5 QA-eligible trials from
+  at least 3 distinct agents (baselines and probes never count). Below the
+  bar the QA trial still classifies every trial and writes trajectory
+  summaries, but the task completes without a verdict instead of guessing
+  from a handful of runs.
+
+### Fixed
+
+- A below-evidence-bar QA run could never succeed: the brief told the model
+  to omit the `verdict` key while the trial verifier required it, so every
+  such run failed and retried to exhaustion. The brief now asks for
+  `"verdict": null`. The same import path also fired the GitHub PR-refresh
+  hook twice; it fires once now.
+- The QA brief gave the model two conflicting shapes for
+  `trajectory_summary` and referenced a taxonomy label that no longer
+  exists; there is now one shape (with `schema_version`) and the guidance
+  names real labels.
+- The analysis trial verifier silently skipped JSON validation when its
+  interpreter probe failed; it now runs `python3` unconditionally (the
+  analysis image guarantees it).
+
+## [2026-08-10]
+
+### Changed
+
+- The experiment page now answers the QA question directly: each task row
+  carries an Accepted / Rejected / QA-running chip, and the summary bar adds
+  a "QA verdicts" tile when any task in the experiment ran QA. The task
+  page's non-agent trial section is titled "QA" and labels its chips in
+  words ("QA run", "Pre-trial audit") instead of raw kind strings.
+- QA's conclusion and QA's machinery are now separate surfaces. The verdict
+  chip is clickable and opens the task overview — the full graded set,
+  including trials from other experiments with their "elsewhere" chips.
+  Because QA is task-scoped, a verdict can come from a run that did not
+  cover the viewed experiment's trials; the chip then renders dashed with
+  an "earlier run" tooltip saying how many settled trials here it missed.
+  The "(qa report)" shadow experiment is a debug surface: the hop links on
+  the experiment header and the dashboard chip are now admin-only, and its
+  page labels QA/audit trials as their own "QA run" / "Pre-trial audit"
+  columns instead of mirroring the agent matrix.
+
+### Fixed
+
+- Analysis (QA + the pre-trial audit) runs on `fireworks/glm-5p2` instead of
+  Claude via Bedrock. One model everywhere, prod included; no preview split.
+- A frontend-only push to a backend PR took the preview API down: preparing
+  the preview database rotates the branch DB password on every run, but the
+  Modal secret only got republished when a backend deploy followed. The
+  secret now republishes on every prepare run.
+- The per-experiment "(qa report)" shadow experiment could become a task's
+  primary experiment in API responses and the CLI submit summary, because
+  the eager pre-trial audit linked it first. Shadow experiments are now
+  excluded from primary-experiment picks and task-row experiment chips.
+
+### Removed
+
+- The `oddish report` CLI command. The reports feature it drove was removed
+  with the analyzer tables; the command only hit the deleted `POST /reports`
+  endpoint.
+
 ## [2026-08-07]
 
 ### Changed
 
+- Every analysis job now runs as a trial, distinguished by `trials.kind`:
+  one `qa` trial per task (classification + trajectory summaries + verdict)
+  and one `audit` trial per task version at sweep time. Analysis trials run
+  on their own task image, produce a JSON artifact that a verifier checks,
+  and live in a per-experiment "(qa report)" shadow experiment linked from
+  the dashboard. Settlement importers write the same columns as before, so
+  dashboards, GitHub comments, and alerts are unchanged. Non-'agent' kinds
+  are excluded from quotas, the leaderboard, alerts, facets, queue scans,
+  and public shares; their spend stays visible in admin views.
 - The verdict now says `accept` or `reject` instead of `is_good: true/false`. Stored payloads keep `is_good` too, so old rows, the dashboard queries, and the Slack alert still work. The badge shows "Accepted" or "Rejected".
 - The verdict judge used to bury its hard rules inside exceptions, and it accepted a task whose own audit had found a `must_fix` leak — on tests the untouched base model already passed (0.96 against a 0.25 threshold). The prompt (`verdict_prompt.txt`) is rewritten as two steps: first look for evidence that rejects the task by itself (a leak, weak tests, a failed baseline), and only then weigh the trials' opinions, which need agreement.
 - The task overview panel used to list only the current experiment's trials, but the verdict is computed over every trial of the task — so the panel could show a verdict whose deciding trial it refused to list. It now shows every trial of the version. Trials from other experiments carry a dashed "elsewhere" chip and open in a new tab. Long subtypes also stopped pushing the "View trial" button out of its row.
@@ -18,6 +98,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Removed
 
+- The analyzer reports feature is gone: the `analyzers` / `analyzer_blocks` /
+  `analyzer_experiments` tables (dropped by migration `drop_analyzers_001`),
+  the reports REST endpoints and ops scripts, the dashboard Analyzers tab,
+  and the report evals package. Nobody used it.
+- The block framework, worker-local classifier/verdict/pre-trial/summary
+  execution, the Daytona sandbox runtime, the QA/ANALYZER/ANALYSIS handlers,
+  and the QA orphan-recovery subsystem. Workers no longer hold LLM keys.
+  Removed settings: `ODDISH_VERDICT_MODEL`, `ODDISH_VERDICT_FALLBACK_MODEL`,
+  `ODDISH_PRE_TRIAL_MODEL`, `ODDISH_PRE_TRIAL_TIMEOUT`,
+  `ODDISH_AGENT_DAYTONA_SNAPSHOT`, `ODDISH_ANALYZER_SANDBOX_ENABLED`,
+  `ODDISH_POST_TRIAL_SANDBOX_ENABLED`. `analysis_costs` becomes read-only
+  history. `GET /trials/{id}/trajectory/summary` serves the stored summary
+  only.
 - The cc_chat dashboard chat feature is gone end to end: the `/chat-sessions`
   backend router and orchestrator, the chat drawer/button UI and its
   `/api/chat-sessions` proxies in the frontend, the `ChatSession` /