Skip to content

Commit b3c157d

Browse files
committed
Merge origin/staging into claude/everything-is-a-trial
Staging advanced to 3dc0cf3 and diverged, making the PR un-mergeable (mergeable_state=dirty). That silently stopped GitHub from spawning the PR Preview workflow (a pull_request event needs the merge ref), so the shadow-picker fix never deployed. Resolving the divergence: - QA/audit stay trial-based (this PR's design) over staging's job-based QA/ANALYSIS in every conflict (qa.py, trials.py, queue.py, handlers.py, cleanup.py). Staging's deleted qa_handler / backfill / cleanup tests and the worker-side trajectory summarizer stay deleted. - Reconciled the two parallel verdict-state refactors: staging's oddish.core.verdict_state (pure state-setters, preserves the published verdict) replaces this branch's removed verdict_sync.clear_inflight_verdict; the one stale call now uses the equivalent abandon_verdict. - Restored the or_ sqlalchemy import that staging's baseline-gate deferral (#1051) needs -- the merge kept our import line but staging's usage. - Linearized the Alembic fork: staging's verdict_state_001 now chains onto drop_analyzers_001 (single head) instead of branching off drop_prompt_registry_001 alongside this branch's chain. - Trajectory summary endpoint keeps this branch's read-only form (the QA trial writes the summary); staging's on-demand generator is gone. Both suites match the staging tip's pre-existing failures with zero new ones; the two pr_preview workflow tests this branch adjusted now pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj
2 parents 39b0a03 + 3dc0cf3 commit b3c157d

33 files changed

Lines changed: 1233 additions & 141 deletions
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/usr/bin/env bash
2+
# Delete Supabase preview branches created more than MAX_AGE_DAYS ago. The
3+
# project caps active branch projects at 50 and a PR that is abandoned or
4+
# force-closed never runs the teardown that frees its branch, so the pool drains
5+
# until new PRs cannot provision a preview database at all. Deletion is
6+
# recoverable: the PR's next preview run rebuilds the branch from the prod
7+
# schema snapshot.
8+
set -euo pipefail
9+
10+
: "${SUPABASE_ACCESS_TOKEN:?}"
11+
: "${SUPABASE_PROJECT_REF:?}"
12+
13+
MAX_AGE_DAYS="${MAX_AGE_DAYS:-7}"
14+
DRY_RUN="${DRY_RUN:-false}"
15+
16+
case "$MAX_AGE_DAYS" in
17+
'' | *[!0-9]*)
18+
echo "MAX_AGE_DAYS must be a whole number of days, got '$MAX_AGE_DAYS'" >&2
19+
exit 1
20+
;;
21+
esac
22+
23+
cutoff=$(($(date +%s) - MAX_AGE_DAYS * 86400))
24+
25+
# jq aborts on a branch whose created_at it cannot parse, which fails the whole
26+
# script before any delete. That is deliberate: an unreadable listing must never
27+
# be read as "nothing is stale".
28+
stale=$(supabase branches list --project-ref "$SUPABASE_PROJECT_REF" -o json \
29+
| jq -r --argjson cutoff "$cutoff" '
30+
.[] | select(.persistent != true)
31+
| select(.name | test("^pr-[0-9]+$"))
32+
| select((.created_at | sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601) < $cutoff)
33+
| [.id, .name, .created_at] | @tsv')
34+
35+
if [ -z "$stale" ]; then
36+
echo "no preview branches older than $MAX_AGE_DAYS days"
37+
exit 0
38+
fi
39+
40+
failed=0
41+
while IFS=$'\t' read -r id name created_at; do
42+
if [ "$DRY_RUN" = "true" ]; then
43+
echo "would delete $name ($id, created $created_at)"
44+
continue
45+
fi
46+
echo "deleting $name ($id, created $created_at)"
47+
# </dev/null so the interactive CLI cannot swallow the list this loop reads.
48+
supabase branches delete "$id" --project-ref "$SUPABASE_PROJECT_REF" </dev/null || failed=1
49+
done <<<"$stale"
50+
51+
exit "$failed"

.github/workflows/pr-preview.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,12 @@ jobs:
257257
- prepare-preview-database
258258
- deploy-preview-backend
259259
- update-vercel-preview
260-
if: always() && github.event.action != 'closed'
260+
# `!cancelled()` rather than `always()`: the gate must still run when
261+
# upstream jobs are SKIPPED (fork and promotion paths), but a run that
262+
# cancel-in-progress superseded must not publish a failing check and a
263+
# failing deployment status for a commit whose replacement run is still
264+
# building.
265+
if: "!cancelled() && github.event.action != 'closed'"
261266
runs-on: ubuntu-latest
262267
# No job-level `environment:` key here: GitHub attributes that deployment
263268
# record to the person who triggered the run, so pull requests showed a
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Preview Prune
2+
3+
# Sweeps Supabase preview branches older than a week so the 50-active-branch
4+
# ceiling never blocks a new PR's "Prepare preview database" job. Runs on its
5+
# own schedule rather than at the point of failure, so a PR that hits the
6+
# ceiling is a bug in this sweep, not something to recover from inline.
7+
8+
on:
9+
schedule:
10+
- cron: "0 6 * * *"
11+
workflow_dispatch:
12+
inputs:
13+
max_age_days:
14+
description: "Delete preview branches created more than this many days ago"
15+
default: "7"
16+
dry_run:
17+
description: "Log what would be deleted without deleting it"
18+
type: boolean
19+
default: false
20+
21+
concurrency:
22+
group: preview-prune
23+
cancel-in-progress: false
24+
25+
jobs:
26+
prune:
27+
name: Prune stale preview branches
28+
runs-on: ubuntu-latest
29+
timeout-minutes: 20
30+
permissions:
31+
contents: read
32+
packages: read
33+
container:
34+
image: ghcr.io/abundant-ai/oddish-ci-base:latest
35+
credentials:
36+
username: ${{ github.actor }}
37+
password: ${{ secrets.GITHUB_TOKEN }}
38+
env:
39+
SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_ACCESS_TOKEN }}
40+
SUPABASE_PROJECT_REF: ${{ vars.SUPABASE_PROJECT_REF }}
41+
MAX_AGE_DAYS: ${{ inputs.max_age_days || '7' }}
42+
DRY_RUN: ${{ inputs.dry_run || false }}
43+
steps:
44+
- name: Checkout
45+
uses: actions/checkout@v5
46+
47+
- name: Prune stale Supabase preview branches
48+
run: "$GITHUB_WORKSPACE/.github/scripts/preview/prune_stale_supabase_branches.sh"

AGENTS.md

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,15 @@ High-level flow:
162162
nonterminal trial in the org. Final result settlement performs the same
163163
check for agents without live usage. Cancellation retires queued, running,
164164
blocked, and retrying worker jobs in the database before terminating remote
165-
handles; a task is failed only when no other live trial remains.
165+
handles; a task is failed only when no other live trial remains. If quota
166+
cancellation interrupts a replacement QA pass, the last successful verdict
167+
is restored through `cancel_verdict`; a terminal QA failure instead clears
168+
that preserved payload through `fail_verdict`. All task verdict-column
169+
mutations go through `oddish.core.verdict_state`: a published payload may
170+
coexist with QUEUED/RUNNING while its replacement is active, but it must
171+
return to SUCCESS if that pass is abandoned. The
172+
`ck_tasks_published_verdict_status` database constraint rejects a published
173+
payload with a missing or FAILED status.
166174
6. Trial completion persists queryable execution metrics on the trial row:
167175
input/cache/output tokens, total trajectory steps, native runtime cost when
168176
reported, phase timing, trajectory availability, arbitrary verifier
@@ -222,11 +230,14 @@ a code change that ships with a deploy.
222230
- unified claim/dispatch SQL, one `run_single_worker_job` runner, and a
223231
handler registry (`TrialJobHandler`, `TaskExpandJobHandler`,
224232
`TagProjectJobHandler`)
225-
- analysis trials (`oddish.workers.analysis_trials` / `analyzer_trials`):
226-
brief builders, settlement importers, and the audit/QA/analyzer pipeline
227-
edges. Workers execute no LLM calls of their own (the one exception is the
228-
probe transcript summarizer in `oddish/worker/probe_analysis.py`); every
229-
analysis agent runs as a trial on the analysis model's queue key
233+
- analysis trials (`oddish.workers.analysis_trials`): brief builders,
234+
settlement importers, and the audit/QA pipeline edges. Workers execute no
235+
LLM calls of their own (the one exception is the probe transcript
236+
summarizer in `oddish/worker/probe_analysis.py`); every analysis agent
237+
runs as a trial on the analysis model's queue key
238+
- the verdict state machine (`oddish.core.verdict_state`), the only writer
239+
for `tasks.verdict*` lifecycle columns, which preserves the last published
240+
result until a replacement QA pass succeeds or terminally fails
230241
- shared queue-slot leasing, per-queue-key concurrency limits, and
231242
per-user fairness on `TRIAL` claims
232243
- database-backed admin concurrency overrides; these take precedence over
@@ -472,7 +483,7 @@ call the shared `oddish.core.endpoints.deletion` helpers.
472483

473484
Public share links use 256-bit `public_token` values and are access-by-link, not
474485
enumerable. The unauthenticated `/public/experiments` list intentionally returns
475-
no share tokens. Public task/trial/file routes must stay scoped under
486+
no share tokens. Public task/trial/live/file routes must stay scoped under
476487
`/public/experiments/{public_token}/...` and verify membership in that shared
477488
experiment; do not reintroduce `/public/tasks/{task_id}` or
478489
`/public/trials/{trial_id}` ID-only access. Unpublishing an experiment clears

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
8787

8888
### Fixed
8989

90+
- Quota cancellation, retry, and append reconciliation no longer hide a preserved accepted verdict by leaving its payload paired with a missing status. Verdict lifecycle changes now use one state-transition module: replacement QA retains the published payload while queued/running, cancellation or a no-op restores it to `SUCCESS`, and only terminal QA failure discards it. A database constraint repairs and prevents invalid payload/status pairs.
9091
- Worker heartbeats used to stop as soon as the agent finished, but the worker still had to upload and save the results. When that took over 15 minutes, the cleanup sweep marked the trial "Worker heartbeat stalled for over 15 minutes", threw away the finished result, and re-ran the whole trial. The heartbeat now runs until the results are saved and settled.
9192

9293
---

backend/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,7 @@ All routes require auth unless marked public.
302302
| GET | `/public/experiments/{public_token}/tasks` | Public tasks and trials for a shared experiment |
303303
| GET | `/public/experiments/{public_token}/tasks/{task_id}` | Public task status within a shared experiment |
304304
| GET | `/public/experiments/{public_token}/tasks/{task_id}/trials` | Public trial list within a shared experiment |
305+
| GET | `/public/experiments/{public_token}/trials/{trial_id}/live` | Public live transcript and running usage |
305306
| GET | `/public/experiments/{public_token}/trials/{trial_id}/logs` | Public trial logs |
306307
| GET | `/public/experiments/{public_token}/trials/{trial_id}/logs/structured` | Public structured logs |
307308
| GET | `/public/experiments/{public_token}/trials/{trial_id}/trajectory` | Public trajectory |

backend/tests/test_pr_preview_workflow.py

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1+
import json
12
import os
23
import re
34
import shutil
45
import subprocess
56
import tempfile
7+
from datetime import datetime, timedelta, timezone
68
from pathlib import Path
79

810
import pytest
@@ -11,10 +13,12 @@
1113
REPO = Path(__file__).resolve().parents[2]
1214
WORKFLOW = REPO / ".github/workflows/pr-preview.yml"
1315
RESET_WORKFLOW = REPO / ".github/workflows/preview-reset.yml"
16+
PRUNE_WORKFLOW = REPO / ".github/workflows/preview-prune.yml"
1417
PREVIEW = REPO / ".github/scripts/preview"
1518
PREPARE = PREVIEW / "prepare_preview_database.sh"
1619
COMPUTE_PLAN = PREVIEW / "compute_deployment_plan.sh"
1720
DEPLOY = PREVIEW / "deploy_preview_backend.sh"
21+
PRUNE = PREVIEW / "prune_stale_supabase_branches.sh"
1822
MODAL_APP = REPO / "backend/modal_app.py"
1923

2024
URL_FRAGMENT = "abundant-ai-preview--oddish-pr-{0}-api.modal.run"
@@ -442,3 +446,165 @@ def test_reset_reuses_preview_scripts():
442446
)
443447
assert prepare_step["env"]["DEPLOY_BACKEND"] == "true"
444448
assert prepare_step["env"]["RUN_MIGRATIONS"] == "true"
449+
450+
451+
def _prune_wf():
452+
return yaml.safe_load(PRUNE_WORKFLOW.read_text())
453+
454+
455+
def test_prune_runs_on_a_schedule():
456+
on = _on(_prune_wf())
457+
assert on["schedule"], "prune must run unattended, not only on dispatch"
458+
assert "workflow_dispatch" in on
459+
assert on["workflow_dispatch"]["inputs"]["max_age_days"]["default"] == "7"
460+
461+
462+
def test_prune_workflow_invokes_the_script():
463+
job = _prune_wf()["jobs"]["prune"]
464+
steps = job["steps"]
465+
assert any("prune_stale_supabase_branches.sh" in s.get("run", "") for s in steps)
466+
for key in ("SUPABASE_ACCESS_TOKEN", "SUPABASE_PROJECT_REF", "MAX_AGE_DAYS"):
467+
assert key in job["env"]
468+
# Dispatch inputs reach the script through env, never interpolated into a
469+
# run: body where they would be shell injection.
470+
assert not any("${{" in s.get("run", "") for s in steps)
471+
472+
473+
def test_prune_script_is_executable():
474+
# The workflow runs it by path, so a lost exec bit is a broken cron.
475+
assert os.access(PRUNE, os.X_OK)
476+
477+
478+
def _branch(name, days_old, *, persistent=False, created_at=None):
479+
if created_at is None:
480+
stamp = datetime.now(timezone.utc) - timedelta(days=days_old)
481+
created_at = stamp.strftime("%Y-%m-%dT%H:%M:%SZ")
482+
return {
483+
"id": f"id-{name}",
484+
"name": name,
485+
"persistent": persistent,
486+
"created_at": created_at,
487+
}
488+
489+
490+
def _run_prune(branches, *, env=None, fail_delete=""):
491+
tmp = Path(tempfile.mkdtemp())
492+
bins = tmp / "bin"
493+
bins.mkdir()
494+
listing = tmp / "branches.json"
495+
listing.write_text(json.dumps(branches))
496+
deleted = tmp / "deleted"
497+
deleted.write_text("")
498+
fake = bins / "supabase"
499+
# `delete` drains stdin the way the real interactive CLI does, so a script
500+
# that fed it the branch list would only ever delete the first branch.
501+
fake.write_text(
502+
"#!/usr/bin/env bash\n"
503+
'case "$2" in\n'
504+
f' list) cat "{listing}" ;;\n'
505+
" delete)\n"
506+
" cat >/dev/null\n"
507+
f' echo "$3" >> "{deleted}"\n'
508+
f' [ "$3" = "{fail_delete}" ] && exit 1\n'
509+
" ;;\n"
510+
"esac\n"
511+
"exit 0\n"
512+
)
513+
fake.chmod(0o755)
514+
proc = subprocess.run(
515+
["bash", str(PRUNE)],
516+
env={
517+
**os.environ,
518+
"PATH": f"{bins}:{os.environ['PATH']}",
519+
"SUPABASE_ACCESS_TOKEN": "token",
520+
"SUPABASE_PROJECT_REF": "ref",
521+
**(env or {}),
522+
},
523+
stdin=subprocess.DEVNULL,
524+
capture_output=True,
525+
text=True,
526+
)
527+
return proc, deleted.read_text().split()
528+
529+
530+
@needs_bash
531+
def test_prune_deletes_only_stale_pr_branches():
532+
proc, deleted = _run_prune(
533+
[
534+
_branch("pr-1", 10),
535+
_branch("pr-2", 2),
536+
_branch("pr-3", 30, persistent=True),
537+
_branch("main", 99, persistent=True),
538+
_branch("staging-preview", 99),
539+
]
540+
)
541+
assert proc.returncode == 0, proc.stderr
542+
assert deleted == ["id-pr-1"]
543+
544+
545+
@needs_bash
546+
def test_prune_honours_max_age_days():
547+
proc, deleted = _run_prune(
548+
[_branch("pr-1", 10), _branch("pr-2", 2)],
549+
env={"MAX_AGE_DAYS": "1"},
550+
)
551+
assert proc.returncode == 0, proc.stderr
552+
assert sorted(deleted) == ["id-pr-1", "id-pr-2"]
553+
554+
555+
@needs_bash
556+
def test_prune_dry_run_deletes_nothing():
557+
proc, deleted = _run_prune([_branch("pr-1", 10)], env={"DRY_RUN": "true"})
558+
assert proc.returncode == 0, proc.stderr
559+
assert deleted == []
560+
assert "would delete pr-1" in proc.stdout
561+
562+
563+
@needs_bash
564+
def test_prune_deletes_every_stale_branch():
565+
# Regression: the delete CLI must not consume the loop's branch list.
566+
proc, deleted = _run_prune([_branch(f"pr-{n}", 10) for n in (1, 2, 3)])
567+
assert proc.returncode == 0, proc.stderr
568+
assert sorted(deleted) == ["id-pr-1", "id-pr-2", "id-pr-3"]
569+
570+
571+
@needs_bash
572+
def test_prune_accepts_fractional_second_timestamps():
573+
stamp = datetime.now(timezone.utc) - timedelta(days=10)
574+
proc, deleted = _run_prune(
575+
[_branch("pr-1", 0, created_at=stamp.strftime("%Y-%m-%dT%H:%M:%S.123456Z"))]
576+
)
577+
assert proc.returncode == 0, proc.stderr
578+
assert deleted == ["id-pr-1"]
579+
580+
581+
@needs_bash
582+
def test_prune_fails_closed_on_unreadable_timestamp():
583+
proc, deleted = _run_prune(
584+
[_branch("pr-1", 10), _branch("pr-2", 0, created_at="whenever")]
585+
)
586+
assert proc.returncode != 0
587+
assert deleted == []
588+
589+
590+
@needs_bash
591+
def test_prune_reports_a_failed_delete_and_keeps_going():
592+
proc, deleted = _run_prune(
593+
[_branch("pr-1", 10), _branch("pr-2", 10)], fail_delete="id-pr-1"
594+
)
595+
assert proc.returncode == 1
596+
assert sorted(deleted) == ["id-pr-1", "id-pr-2"]
597+
598+
599+
@needs_bash
600+
def test_prune_rejects_a_non_numeric_age():
601+
proc, deleted = _run_prune([_branch("pr-1", 10)], env={"MAX_AGE_DAYS": "7 days"})
602+
assert proc.returncode == 1
603+
assert deleted == []
604+
605+
606+
@needs_bash
607+
def test_prune_is_quiet_when_nothing_is_stale():
608+
proc, deleted = _run_prune([_branch("pr-1", 2)])
609+
assert proc.returncode == 0, proc.stderr
610+
assert deleted == []

0 commit comments

Comments
 (0)