Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,12 @@ High-level flow:
nonterminal trial in the org. Final result settlement performs the same
check for agents without live usage. Cancellation retires queued, running,
blocked, and retrying worker jobs in the database before terminating remote
handles; a task is failed only when no other live trial remains.
handles; a task is marked CANCELLED only when no other live trial remains.
Task status reflects the execution outcome (CANCELLED = a person or quota
stopped the run; FAILED = the pipeline broke), while the verdict columns
independently keep the judgment — a kept SUCCESS verdict stays valid on a
cancelled task. Cancelling a task in ANALYZING / VERDICT_PENDING (trials
all finished, only QA stopped) completes it instead.
6. Trial completion persists queryable execution metrics on the trial row:
input/cache/output tokens, total trajectory steps, native runtime cost when
reported, phase timing, trajectory availability, arbitrary verifier
Expand Down
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

---

## [2026-08-08]

### Added

- Tasks now have a first-class terminal `cancelled` status. Cancelling an
active run (`oddish cancel`, or a quota cap being reached) marks the task
`cancelled` instead of `failed` — status reflects the execution outcome
(stopped, not broken), while the verdict columns independently keep the
judgment, so a kept accept verdict stays valid on a cancelled task. Failure
metrics and automation keying on `failed` no longer count cancellations.
Cancelling a task whose trials all finished (QA cancel, or `oddish cancel`
during `verdict_pending`) marks it `completed` — only the judgment was
stopped, and the verdict columns record that. Retrying a trial of a
cancelled task resurrects it, exactly like a failed one. The task browse
sidebar gains a "Cancelled" status filter. Core migration
`cancel01_add_cancelled` adds the enum value; pre-existing cancelled rows
keep their old status. ⚠️ Older installed CLIs treat only
`completed`/`failed` as terminal, so `oddish run`/`status --watch`/`pull
--wait` loops keep polling on a task another client cancelled until
interrupted; they render the new status as plain text and do not crash.

## [2026-08-07]

### Changed
Expand Down
5 changes: 5 additions & 0 deletions DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,11 @@ Use `oddish cancel` to stop queued or running work without deleting the task
itself. Completed trials are preserved. By default it cancels all active task
runs; use `--qa` to cancel only the task-level QA job.

A task cancelled mid-run lands in the terminal `cancelled` status (stopped,
not broken — `failed` still means the pipeline broke). A task whose trials
had all finished lands `completed` instead: only the judgment was stopped,
which the verdict shows. A verdict the task already held is kept either way.

```bash
# Cancel all active runs for a task
oddish cancel <task_id>
Expand Down
2 changes: 1 addition & 1 deletion backend/preview_seed.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
_MAX_BIND_PARAMS = 28000
_LOAD_STREAMS = 6

_TERMINAL_TASK_STATUSES = ("COMPLETED", "FAILED")
_TERMINAL_TASK_STATUSES = ("COMPLETED", "FAILED", "CANCELLED")
_TERMINAL_TRIAL_STATUSES = ("SUCCESS", "FAILED")
_TERMINAL_JOB_STATUSES = ("SUCCESS", "FAILED", "CANCELLED")

Expand Down
1 change: 1 addition & 0 deletions frontend/src/lib/tasks-filters.ts
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ const STATUS_OPTIONS: Option[] = [
{ value: "ANALYZING", label: "Analyzing" },
{ value: "VERDICT_PENDING", label: "Verdict pending" },
{ value: "FAILED", label: "Failed" },
{ value: "CANCELLED", label: "Cancelled" },
];

const PRIORITY_OPTIONS: Option[] = [
Expand Down
3 changes: 2 additions & 1 deletion frontend/src/lib/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ type TaskStatus =
| "analyzing"
| "verdict_pending"
| "completed"
| "failed";
| "failed"
| "cancelled";

type TrialStatus =
| "pending"
Expand Down
40 changes: 40 additions & 0 deletions oddish/alembic/versions/cancel01_add_cancelled_task_status.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
"""add CANCELLED to the taskstatus enum (task status)

A user- or quota-cancelled run becomes a first-class ``CANCELLED`` task status
instead of ``FAILED`` (or ``COMPLETED`` when a kept verdict forced the
COMPLETED override). Status now reflects the execution outcome — stopped, not
broken — while the verdict columns independently keep the judgment, so a kept
SUCCESS verdict stays valid on a cancelled task.

The ``taskstatus`` PG enum uses the enum *member names* (uppercase) as labels
(PENDING/RUNNING/.../FAILED), so we add ``'CANCELLED'``. No backfill:
pre-existing cancelled rows stay ``FAILED`` + the cancel message.

``ALTER TYPE ... ADD VALUE`` MUST run outside a transaction, and the new value
MUST NOT be used in the same migration. Mirrors skip01 / qa01 precedents.

Revision ID: cancel01_add_cancelled
Revises: drop_prompt_registry_001
Create Date: 2026-08-08 00:00:00.000000
"""

from typing import Sequence, Union

from alembic import op


revision: str = "cancel01_add_cancelled"
down_revision: Union[str, Sequence[str], None] = "drop_prompt_registry_001"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
with op.get_context().autocommit_block():
op.execute("ALTER TYPE taskstatus ADD VALUE IF NOT EXISTS 'CANCELLED'")


def downgrade() -> None:
# Postgres can't cleanly drop enum values; 'CANCELLED' stays after
# downgrade. Mirrors every other enum-add migration.
pass
19 changes: 15 additions & 4 deletions oddish/src/oddish/cli/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -2002,6 +2002,7 @@ def format_task_status(status: str) -> str:
"verdict_pending": ("magenta", "verdict"),
"completed": ("green", "completed"),
"failed": ("red", "failed"),
"cancelled": ("yellow", "cancelled"),
}
style, label = style_map.get(status.lower(), ("white", status))
return f"[{style}]{label}[/{style}]"
Expand Down Expand Up @@ -2082,7 +2083,9 @@ def format_verdict_status(verdict_status: str) -> str:

def _summarize_experiment_tasks(tasks: list[dict]) -> dict:
total_tasks = len(tasks)
task_completed = sum(1 for t in tasks if t.get("status") in ("completed", "failed"))
task_completed = sum(
1 for t in tasks if t.get("status") in ("completed", "failed", "cancelled")
)
task_running = sum(1 for t in tasks if t.get("status") == "running")
task_pending = total_tasks - task_completed - task_running

Expand Down Expand Up @@ -2418,7 +2421,10 @@ def watch_experiment(api_url: str, experiment_id: str) -> None:

live.update(_build_experiment_table(experiment_id, tasks))

if all(t.get("status") in ("completed", "failed") for t in tasks):
if all(
t.get("status") in ("completed", "failed", "cancelled")
for t in tasks
):
break

time.sleep(2)
Expand Down Expand Up @@ -2617,7 +2623,12 @@ def watch_task(
table.add_row("", ", ".join(summary_parts), "", "", "", "")

# Show verdict status if in later pipeline stages
if task_status in ("analyzing", "verdict_pending", "completed"):
if task_status in (
"analyzing",
"verdict_pending",
"completed",
"cancelled",
):
verdict_status = result.get("verdict_status")
if verdict_status:
verdict_display = {
Expand All @@ -2638,7 +2649,7 @@ def watch_task(
t.get("status") in terminal for t in all_trials
):
break
elif task_status in ("completed", "failed"):
elif task_status in ("completed", "failed", "cancelled"):
break

time.sleep(2)
Expand Down
4 changes: 2 additions & 2 deletions oddish/src/oddish/cli/pull.py
Original file line number Diff line number Diff line change
Expand Up @@ -581,14 +581,14 @@ def _is_task_terminal(client: httpx.Client, task_id: str) -> bool:
task = _get_task_status(client, task_id)
if not task:
return False
return task.get("status") in ("completed", "failed")
return task.get("status") in ("completed", "failed", "cancelled")


def _is_experiment_terminal(client: httpx.Client, experiment_id: str) -> bool:
tasks = _list_tasks_for_experiment(client, experiment_id)
if not tasks:
return True
return all(t.get("status") in ("completed", "failed") for t in tasks)
return all(t.get("status") in ("completed", "failed", "cancelled") for t in tasks)


def _pull_once(
Expand Down
2 changes: 1 addition & 1 deletion oddish/src/oddish/cli/status.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def _status_json(
entry["tasks"] += 1
if task.get("status") == "running":
entry["running"] += 1
if task.get("status") in ("completed", "failed"):
if task.get("status") in ("completed", "failed", "cancelled"):
entry["done"] += 1
entry["total_trials"] += task.get("total", 0) or 0
entry["completed_trials"] += task.get("completed", 0) or 0
Expand Down
11 changes: 4 additions & 7 deletions oddish/src/oddish/core/endpoints/qa.py
Original file line number Diff line number Diff line change
Expand Up @@ -188,13 +188,10 @@ async def cancel_task_qa_core(
trial.analysis_error = USER_CANCELLED_MESSAGE
trial.analysis_finished_at = now_value
if task.status == TaskStatus.VERDICT_PENDING:
# A restored verdict means the task is judged; only a task with
# no verdict fails on cancel.
task.status = (
TaskStatus.COMPLETED
if task.verdict_status == VerdictStatus.SUCCESS
else TaskStatus.FAILED
)
# VERDICT_PENDING means every trial already finished — cancelling
# QA stops only the judgment, so the run itself completed. The
# verdict columns carry the cancelled (or kept) judgment.
task.status = TaskStatus.COMPLETED
task.finished_at = now_value
# The pre-trial audit runs inside a QA job (full or audit-only). A request
# left QUEUED (or a claim left RUNNING) with no job behind it would keep
Expand Down
1 change: 1 addition & 0 deletions oddish/src/oddish/core/endpoints/trials.py
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,7 @@ async def retry_trial_core(
TaskStatus.VERDICT_PENDING,
TaskStatus.COMPLETED,
TaskStatus.FAILED,
TaskStatus.CANCELLED,
):
task.status = TaskStatus.RUNNING
task.finished_at = None
Expand Down
5 changes: 5 additions & 0 deletions oddish/src/oddish/core/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -569,7 +569,12 @@ def resolve_task_status(
non-pass). SKIPPED is terminal — the trial never ran — so it counts toward
"done" alongside completed/failed; otherwise a task with gate-skipped trials
would never resolve to COMPLETED.

CANCELLED is sticky: a cancelled run's trials are all terminal (the cancel
finalized them), which must not read as "all trials finished" = COMPLETED.
"""
if task.status is TaskStatus.CANCELLED:
return task.status
if total > 0 and completed + failed + skipped >= total:
return TaskStatus.COMPLETED
return task.status
Expand Down
1 change: 1 addition & 0 deletions oddish/src/oddish/core/ingest/trial_imports.py
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,7 @@ async def initialize_trial_import(
elif task.status in (
TaskStatus.COMPLETED,
TaskStatus.FAILED,
TaskStatus.CANCELLED,
TaskStatus.ANALYZING,
TaskStatus.VERDICT_PENDING,
):
Expand Down
4 changes: 3 additions & 1 deletion oddish/src/oddish/core/quota_enforcement.py
Original file line number Diff line number Diff line change
Expand Up @@ -342,7 +342,9 @@ async def _reconcile_cancelled_tasks(
if task.id not in exhausted:
continue
if task.status in _ACTIVE_TASK_STATUSES:
task.status = TaskStatus.FAILED
# Quota stopped the run — it didn't break. The verdict columns
# below still record the cancelled judgment independently.
task.status = TaskStatus.CANCELLED
task.finished_at = now
tasks_cancelled += 1
if task.verdict_status in (
Expand Down
1 change: 1 addition & 0 deletions oddish/src/oddish/db/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ class TaskStatus(str, Enum):
VERDICT_PENDING = "verdict_pending" # All analyses done, verdict running
COMPLETED = "completed" # All stages complete
FAILED = "failed" # Terminal failure
CANCELLED = "cancelled" # Terminal: stopped by a person or quota, not broken


class JobStatus(str, Enum):
Expand Down
19 changes: 11 additions & 8 deletions oddish/src/oddish/queue.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,23 +275,25 @@ async def cancel_tasks_runs(
tasks_cancelled = 0
for task in tasks:
task_updated = False
failed_by_this_cancel = False
if task.status in ACTIVE_TASK_STATUSES:
task.status = TaskStatus.FAILED
# Status reflects the execution outcome; the verdict columns keep
# the judgment independently (a kept SUCCESS verdict stays valid
# on a cancelled task). In ANALYZING / VERDICT_PENDING every trial
# already finished — only QA was stopped — so the run itself
# completed (same rule as the QA cancel endpoint).
task.status = (
TaskStatus.COMPLETED
if task.status in (TaskStatus.ANALYZING, TaskStatus.VERDICT_PENDING)
else TaskStatus.CANCELLED
)
task.finished_at = now
task_updated = True
failed_by_this_cancel = True
if (
task.id in canceled_verdict_task_ids
or task.verdict_status in ACTIVE_PIPELINE_STATUSES
):
cancel_verdict(task, error=USER_CANCELLED_MESSAGE, now=now)
task_updated = True
# A task that still holds a successful verdict is judged. Cancelling
# its extra trials completes it; it must not read as a failed task
# with an accepted verdict (same rule as the QA cancel endpoint).
if failed_by_this_cancel and task.verdict_status == VerdictStatus.SUCCESS:
task.status = TaskStatus.COMPLETED
if task_updated:
tasks_cancelled += 1

Expand Down Expand Up @@ -1312,6 +1314,7 @@ async def append_trials_to_task(
if new_trials and task.status in (
TaskStatus.COMPLETED,
TaskStatus.FAILED,
TaskStatus.CANCELLED,
TaskStatus.ANALYZING,
TaskStatus.VERDICT_PENDING,
):
Expand Down
4 changes: 2 additions & 2 deletions oddish/src/oddish/workers/queue/cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -1237,7 +1237,7 @@ async def _reset_orphaned_trial_analysis(session) -> tuple[int, int]:
LIKE :gate_skip_pattern
OR t.deleted_at IS NOT NULL
OR (
t.status IN ('COMPLETED', 'FAILED')
t.status IN ('COMPLETED', 'FAILED', 'CANCELLED')
AND NOT EXISTS (
SELECT 1
FROM worker_jobs wj
Expand Down Expand Up @@ -1289,7 +1289,7 @@ async def _reset_orphaned_trial_analysis(session) -> tuple[int, int]:
AND COALESCE(tr.analysis_started_at, tr.updated_at)
< NOW() - make_interval(mins => :stale_minutes)
AND t.deleted_at IS NULL
AND t.status NOT IN ('COMPLETED', 'FAILED')
AND t.status NOT IN ('COMPLETED', 'FAILED', 'CANCELLED')
AND NOT EXISTS (
SELECT 1
FROM worker_jobs wj
Expand Down
2 changes: 1 addition & 1 deletion oddish/src/oddish/workers/queue/trial_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -942,7 +942,7 @@ async def _run_post_trial_hooks(trial_id: str) -> None:
or trial.harbor_stage == "cancelled"
):
return
if task is None or task.status == TaskStatus.FAILED:
if task is None or task.status in (TaskStatus.FAILED, TaskStatus.CANCELLED):
return

await maybe_gate_llm_trials(session, trial_id)
Expand Down
Loading
Loading