feat(admin): exclude models and experiments from spend, and label what's left

Some spend Oddish records was never actually paid for: provider-sponsored
capacity, free preview tiers, vendor credits, a bake-off someone comped.
Operators can now declare it along two axes, from the admin panel or the CLI.

Replaces the LLM-key exclusion list, which keyed off a hash nobody could read
back and only covered the "we have a free key" case. The two axes people
actually reach for:

* Model (`cost_excluded_models`) -- global and retroactive. Every trial that
  ever ran on it stops counting, because the reason its spend isn't real is a
  property of the model.
* Experiment (`cost_excluded_experiments`) -- scoped to trials homed in it, so
  a collection that merely gathers other experiments' trials can't launder
  their cost.

Both fold into `first_party_spend_filter` and the quota inflight predicates,
so excluded spend leaves the admin cost dashboards and stops counting against
caps together, and re-including it restores every dollar.

Dropped from accounting, but deliberately not hidden. Experiment, task, and
trial surfaces still price the work that ran and now mark it as not real, so a
reader reconciling an experiment tile against the admin dashboard finds an
explanation instead of a silent gap. The experiment rollup splits its existing
groups on the exclusion predicate to report `excluded_cost_usd`; trials carry
`cost_exclusion_reason`, which is null both for real spend and for callers
that never resolved exclusions -- absence is not a claim that the money was
real.

`core/cost_exclusions.py` owns the SQL predicates and the `CostExclusions`
Python twin used to label already-loaded rows; keep the two in step.

`trials.llm_key_hash` is left in place and is now write-only -- dropping a
column off the hot trials table is a separate, riskier change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017zmdrcCqnkVM6hWjhPu2T9

Author: claude

AGENTS.md

@@ -334,12 +334,28 @@ status, queue health, worker, orphan, cost, per-user cost, and task-expansion
 handlers must pass `auth.org_id`; never accept an organization selector from
 the client. A user cost drilldown returns 404 when the requested user belongs
 to another org. Deployment-wide diagnostics or mutations (global queue
-status/health and slot topology, model concurrency, shared-channel Slack alert
-settings, and the global cost-excluded LLM-key list) additionally require the active org to match
+status/health and slot topology, model concurrency, shared-channel Slack
+alert settings, and the global cost-exclusion lists) additionally require the
+active org to match
 `ODDISH_OPERATOR_ORG_ID`, which fails closed when unset; the frontend discovers
 that capability through `GET /admin/operator-access` and hides those controls
 for other orgs.
 
+Admin cost exclusions (`oddish/core/cost_exclusions.py`) name spend that was
+never really paid for, along two axes: a **model** (`cost_excluded_models`,
+matched against `trials.model`, global and retroactive) and an **experiment**
+(`cost_excluded_experiments`, matched against `trials.experiment_id` so a
+collection cannot launder gathered trials' cost). Both fold into
+`first_party_spend_filter` and the quota inflight predicates, so excluded
+spend leaves the cost dashboards and stops counting against caps together.
+It is dropped from accounting but **not** hidden: experiment, task, and trial
+surfaces still render the money and label it, via `excluded_cost_usd` on the
+experiment rollup and `cost_exclusion_reason` on `TrialResponse`. Keep the SQL
+predicates and the `CostExclusions` Python twin in step — a surface that
+labels spend differently from the way accounting drops it is worse than one
+that says nothing. Callers that do not pass an exclusions snapshot report
+`cost_exclusion_reason=None`, which means "unresolved", not "real".
+
 The authenticated org-scoped cost leaderboard is served by `GET /leaderboard` in
 `backend/api/routers/dashboard.py`. It shares the admin cost dashboard's
 settled first-party spend basis and must stay in sync with its per-user rows:

DOCS.md

@@ -26,6 +26,7 @@ export ODDISH_API_KEY="ok_..."
 - `oddish cancel` - stop in-flight task runs or task-level QA jobs
 - `oddish backfill-analysis` - (re)run trial analysis for a trial, task, or experiment
 - `oddish costs` - view billable-spend accounting (org-wide, or per-user with `--user`)
+- `oddish cost-exclusions` - manage the models and experiments whose spend doesn't count
 - `oddish pull` - download logs and artifacts
 - `oddish combine` - merge several experiments into a new one
 - `oddish collect` - gather trials from tasks/trial IDs into a shareable read-only collection
@@ -434,6 +435,51 @@ Options
 - `--api TEXT` - Override the API URL
 - `--json` - Emit the raw cost breakdown JSON
 
+## Spend That Doesn't Count
+
+Some spend Oddish records was never actually paid for: provider-sponsored
+capacity, free preview tiers, vendor credits, a bake-off someone comped.
+Operators mark it with `oddish cost-exclusions`, along two axes:
+
+- **Models** - every trial that ever ran on the model stops counting, because
+  the reason its spend isn't real is a property of the model.
+- **Experiments** - only trials the experiment ran *itself* stop counting.
+  Trials it merely gathered from elsewhere keep counting on the experiment that
+  actually spent the money.
+
+Excluded spend is dropped from the admin cost dashboards and from quota
+enforcement. It is **not** hidden: experiment, task, and trial pages still show
+the money, marked as not real, so the two surfaces never disagree silently.
+
+Both lists are deployment-wide and apply retroactively - adding an entry removes
+spend already recorded, and removing one restores every dollar. Operator-only on
+hosted Oddish (a full-scope API key belonging to the operator org); not
+available on a self-hosted core server.
+
+```bash
+# What currently doesn't count
+oddish cost-exclusions list
+oddish cost-exclusions list --kind model --json
+
+# Stop counting a free model, and a comped experiment
+oddish cost-exclusions add model xai/grok-4 --label "sponsored"
+oddish cost-exclusions add experiment "glm sweep" --label "comped"
+
+# Put the spend back (by row id, model name, or experiment name/id)
+oddish cost-exclusions remove model xai/grok-4
+oddish cost-exclusions remove experiment exp_01j...
+```
+
+Options
+
+- `--kind TEXT` - On `list`, limit to one axis: `model` or `experiment`
+- `--label TEXT` - On `add`, why it doesn't count (e.g. `sponsored`)
+- `--api TEXT` - Override the API URL
+- `--json` - Emit raw JSON
+
+An experiment can be named or referenced by id. Names are not unique, so an
+ambiguous name is rejected rather than resolved to a guess - pass the id.
+
 ## Download Outputs
 
 Use `oddish pull` to download logs and artifacts from Oddish to local files.

backend/api/app.py

@@ -245,7 +245,8 @@ def create_app() -> FastAPI:
         api_keys,
         byok,
         clerk_webhooks,
-        cost_excluded_keys,
+        cost_excluded_experiments,
+        cost_excluded_models,
         dashboard,
         documents,
         github_linkage,
@@ -287,7 +288,8 @@ def create_app() -> FastAPI:
     api.include_router(public_analysis.router)
     api.include_router(slack.router)
     api.include_router(admin.router)
-    api.include_router(cost_excluded_keys.router)
+    api.include_router(cost_excluded_models.router)
+    api.include_router(cost_excluded_experiments.router)
     api.include_router(model_display_names.router)
     api.include_router(tags.router)
     api.include_router(reports.router)

backend/api/routers/cost_excluded_experiments.py

@@ -0,0 +1,187 @@
+"""Admin API for excluding an experiment's spend from cost accounting.
+
+Operator-only and deployment-wide, like the model list. Scoped to trials
+**homed** in the experiment, so a collection that merely gathers other
+experiments' trials cannot launder their cost.
+"""
+
+from __future__ import annotations
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.exc import IntegrityError, ProgrammingError
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from auth import AuthContext, can_manage_api_keys, require_admin
+from auth.permissions import require_operator_org
+from oddish.db import (
+    CostExcludedExperimentModel,
+    ExperimentModel,
+    get_session,
+    utcnow,
+)
+from pg_errors import is_undefined_table_error
+
+router = APIRouter(prefix="/admin/cost-excluded-experiments", tags=["Admin"])
+
+
+class CostExcludedExperimentResponse(BaseModel):
+    id: str
+    experiment_id: str
+    experiment_name: str
+    label: str
+    created_by: str | None
+    created_at: str
+
+
+class CreateCostExcludedExperimentRequest(BaseModel):
+    experiment: str
+    label: str = ""
+
+
+def _response(row: CostExcludedExperimentModel) -> CostExcludedExperimentResponse:
+    return CostExcludedExperimentResponse(
+        id=row.id,
+        experiment_id=row.experiment_id,
+        experiment_name=row.experiment_name,
+        label=row.label,
+        created_by=row.created_by_user_id,
+        created_at=row.created_at.isoformat(),
+    )
+
+
+def _require_manage(auth: AuthContext) -> None:
+    require_operator_org(auth)
+    if not can_manage_api_keys(auth):
+        raise HTTPException(
+            status_code=403,
+            detail="Only organization admins may edit the cost-exclusion list",
+        )
+
+
+def _unavailable(exc: ProgrammingError) -> HTTPException:
+    if is_undefined_table_error(exc):
+        return HTTPException(
+            503,
+            "Cost exclusions are not available yet (schema is still "
+            "migrating). Try again shortly.",
+        )
+    raise exc
+
+
+async def _resolve_experiment(session: AsyncSession, ref: str) -> ExperimentModel:
+    """The experiment an operator meant, by id or by name.
+
+    An exact id wins, and resolves with ``include_deleted``: spend from a
+    soft-deleted experiment still shows on cost surfaces, so it must stay
+    excludable. Names resolve among live experiments only and must be
+    unambiguous -- experiment names are not unique.
+    """
+    by_id = await session.scalars(
+        select(ExperimentModel)
+        .where(ExperimentModel.id == ref)
+        .execution_options(include_deleted=True)
+    )
+    experiment = by_id.first()
+    if experiment is not None:
+        return experiment
+    by_name = await session.scalars(
+        select(ExperimentModel).where(ExperimentModel.name == ref).limit(2)
+    )
+    matches = by_name.all()
+    if len(matches) > 1:
+        raise HTTPException(
+            status_code=409,
+            detail="experiment name is ambiguous; use the experiment id",
+        )
+    if not matches:
+        raise HTTPException(status_code=404, detail="experiment not found")
+    return matches[0]
+
+
+@router.get("", response_model=list[CostExcludedExperimentResponse])
+async def list_cost_excluded_experiments(
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> list[CostExcludedExperimentResponse]:
+    require_operator_org(auth)
+    try:
+        async with get_session() as session:
+            rows = await session.scalars(
+                select(CostExcludedExperimentModel).order_by(
+                    CostExcludedExperimentModel.created_at.desc()
+                )
+            )
+            return [_response(row) for row in rows]
+    except ProgrammingError as exc:
+        raise _unavailable(exc)
+
+
+@router.post("", response_model=CostExcludedExperimentResponse)
+async def add_cost_excluded_experiment(
+    request: CreateCostExcludedExperimentRequest,
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> CostExcludedExperimentResponse:
+    _require_manage(auth)
+
+    ref = request.experiment.strip()
+    if not ref:
+        raise HTTPException(status_code=400, detail="experiment must not be empty")
+
+    try:
+        async with get_session() as session:
+            experiment = await _resolve_experiment(session, ref)
+            existing = await session.scalars(
+                select(CostExcludedExperimentModel).where(
+                    CostExcludedExperimentModel.experiment_id == experiment.id
+                )
+            )
+            if existing.first() is not None:
+                raise HTTPException(
+                    status_code=409, detail="experiment is already excluded"
+                )
+
+            row = CostExcludedExperimentModel(
+                experiment_id=experiment.id,
+                # A display snapshot: the row outlives the experiment.
+                experiment_name=experiment.name,
+                label=request.label.strip(),
+                created_by_user_id=auth.user_id,
+            )
+            session.add(row)
+            try:
+                await session.commit()
+            except IntegrityError:
+                raise HTTPException(
+                    status_code=409, detail="experiment is already excluded"
+                )
+            return _response(row)
+    except ProgrammingError as exc:
+        raise _unavailable(exc)
+
+
+@router.delete("/{row_id}")
+async def remove_cost_excluded_experiment(
+    row_id: str,
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> dict:
+    _require_manage(auth)
+    try:
+        async with get_session() as session:
+            result = await session.scalars(
+                select(CostExcludedExperimentModel).where(
+                    CostExcludedExperimentModel.id == row_id
+                )
+            )
+            row = result.first()
+            if row is None:
+                raise HTTPException(
+                    status_code=404, detail="cost-excluded experiment not found"
+                )
+            row.deleted_at = utcnow()
+            await session.commit()
+    except ProgrammingError as exc:
+        raise _unavailable(exc)
+    return {"deleted": row_id}