feat(admin): exclude models and experiments from spend, and label what's left

AGENTS.md

@@ -334,12 +334,28 @@ status, queue health, worker, orphan, cost, per-user cost, and task-expansion
 handlers must pass `auth.org_id`; never accept an organization selector from
 the client. A user cost drilldown returns 404 when the requested user belongs
 to another org. Deployment-wide diagnostics or mutations (global queue
-status/health and slot topology, model concurrency, shared-channel Slack alert
-settings, and the global cost-excluded LLM-key list) additionally require the active org to match
+status/health and slot topology, model concurrency, shared-channel Slack
+alert settings, and the global cost-exclusion lists) additionally require the
+active org to match
 `ODDISH_OPERATOR_ORG_ID`, which fails closed when unset; the frontend discovers
 that capability through `GET /admin/operator-access` and hides those controls
 for other orgs.
 
+Admin cost exclusions (`oddish/core/cost_exclusions.py`) name spend that was
+never really paid for, along two axes: a **model** (`cost_excluded_models`,
+matched against `trials.model`, global and retroactive) and an **experiment**
+(`cost_excluded_experiments`, matched against `trials.experiment_id` so a
+collection cannot launder gathered trials' cost). Both fold into
+`first_party_spend_filter` and the quota inflight predicates, so excluded
+spend leaves the cost dashboards and stops counting against caps together.
+It is dropped from accounting but **not** hidden: experiment, task, and trial
+surfaces still render the money and label it, via `excluded_cost_usd` on the
+experiment rollup and `cost_exclusion_reason` on `TrialResponse`. Keep the SQL
+predicates and the `CostExclusions` Python twin in step — a surface that
+labels spend differently from the way accounting drops it is worse than one
+that says nothing. Callers that do not pass an exclusions snapshot report
+`cost_exclusion_reason=None`, which means "unresolved", not "real".
+
 The authenticated org-scoped cost leaderboard is served by `GET /leaderboard` in
 `backend/api/routers/dashboard.py`. It shares the admin cost dashboard's
 settled first-party spend basis and must stay in sync with its per-user rows:

DOCS.md

@@ -26,6 +26,7 @@ export ODDISH_API_KEY="ok_..."
 - `oddish cancel` - stop in-flight task runs or task-level QA jobs
 - `oddish backfill-analysis` - (re)run trial analysis for a trial, task, or experiment
 - `oddish costs` - view billable-spend accounting (org-wide, or per-user with `--user`)
+- `oddish cost-exclusions` - hide spend for models and experiments that were never really paid for
 - `oddish pull` - download logs and artifacts
 - `oddish combine` - merge several experiments into a new one
 - `oddish collect` - gather trials from tasks/trial IDs into a shareable read-only collection
@@ -434,6 +435,50 @@ Options
 - `--api TEXT` - Override the API URL
 - `--json` - Emit the raw cost breakdown JSON
 
+## Remove Spend Tracking
+
+Hide spend that was never really paid for - sponsored capacity, free preview
+tiers, vendor credits, a comped run. Excluded spend drops off the admin cost
+dashboards and stops counting against quotas. It is still shown on experiment,
+task, and trial pages, marked as not real, so the two never disagree silently.
+
+- **Models** - every trial that used the model stops counting.
+- **Experiments** - trials the experiment ran itself stop counting. Trials it
+  gathered from elsewhere keep counting on the experiment that ran them.
+
+Both lists are deployment-wide and retroactive: adding an entry removes spend
+already recorded, removing one puts every dollar back. Operator-only on hosted
+Oddish (a full-scope API key in the operator org); not available on a
+self-hosted core server. Also editable in the admin dashboard under
+Costs -> Remove Spend Tracking.
+
+```bash
+# What currently doesn't count
+oddish cost-exclusions list
+oddish cost-exclusions list --kind model --json
+
+# Stop counting a free model, and a comped experiment
+oddish cost-exclusions add model kimi-k2 --label "sponsored"
+oddish cost-exclusions add experiment "glm sweep" --label "comped"
+
+# Put the spend back (by row id, model name, or experiment name/id)
+oddish cost-exclusions remove model kimi-k2
+oddish cost-exclusions remove experiment exp_01j...
+```
+
+Options
+
+- `--kind TEXT` - On `list`, limit to one axis: `model` or `experiment`
+- `--label TEXT` - On `add`, why it doesn't count (e.g. `sponsored`)
+- `--api TEXT` - Override the API URL
+- `--json` - Emit raw JSON
+
+`add model` matches what trials actually store, so `kimi-k2` finds trials saved
+as `moonshot/kimi-k2`. A model no trial has ever used is rejected rather than
+saved as an entry that matches nothing. Experiments take a name or an id;
+ambiguous names are rejected, and a collection is rejected because it runs no
+trials of its own.
+
 ## Download Outputs
 
 Use `oddish pull` to download logs and artifacts from Oddish to local files.

backend/api/app.py

@@ -245,7 +245,8 @@ def create_app() -> FastAPI:
         api_keys,
         byok,
         clerk_webhooks,
-        cost_excluded_keys,
+        cost_excluded_experiments,
+        cost_excluded_models,
         dashboard,
         documents,
         github_linkage,
@@ -287,7 +288,8 @@ def create_app() -> FastAPI:
     api.include_router(public_analysis.router)
     api.include_router(slack.router)
     api.include_router(admin.router)
-    api.include_router(cost_excluded_keys.router)
+    api.include_router(cost_excluded_models.router)
+    api.include_router(cost_excluded_experiments.router)
     api.include_router(model_display_names.router)
     api.include_router(tags.router)
     api.include_router(reports.router)

backend/api/routers/cost_excluded_experiments.py

@@ -0,0 +1,151 @@
+from __future__ import annotations
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.exc import IntegrityError, ProgrammingError
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from api.routers.cost_exclusions_shared import soft_delete, unavailable
+from auth import AuthContext, require_admin
+from auth.permissions import require_operator_org
+from oddish.db import CostExcludedExperimentModel, ExperimentModel, get_session
+
+router = APIRouter(prefix="/admin/cost-excluded-experiments", tags=["Admin"])
+
+
+class CostExcludedExperimentResponse(BaseModel):
+    id: str
+    experiment_id: str
+    experiment_name: str
+    label: str
+    created_by: str | None
+    created_at: str
+
+
+class CreateCostExcludedExperimentRequest(BaseModel):
+    experiment: str
+    label: str = ""
+
+
+def _response(row: CostExcludedExperimentModel) -> CostExcludedExperimentResponse:
+    return CostExcludedExperimentResponse(
+        id=row.id,
+        experiment_id=row.experiment_id,
+        experiment_name=row.experiment_name,
+        label=row.label,
+        created_by=row.created_by_user_id,
+        created_at=row.created_at.isoformat(),
+    )
+
+
+async def _resolve_experiment(session: AsyncSession, ref: str) -> ExperimentModel:
+    by_id = await session.scalars(
+        select(ExperimentModel)
+        .where(ExperimentModel.id == ref)
+        .execution_options(include_deleted=True)
+    )
+    experiment = by_id.first()
+    if experiment is not None:
+        return experiment
+    by_name = await session.scalars(
+        select(ExperimentModel).where(ExperimentModel.name == ref).limit(2)
+    )
+    matches = by_name.all()
+    if len(matches) > 1:
+        raise HTTPException(
+            status_code=409,
+            detail="experiment name is ambiguous; use the experiment id",
+        )
+    if not matches:
+        raise HTTPException(status_code=404, detail="experiment not found")
+    return matches[0]
+
+
+@router.get("", response_model=list[CostExcludedExperimentResponse])
+async def list_cost_excluded_experiments(
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> list[CostExcludedExperimentResponse]:
+    require_operator_org(auth)
+    try:
+        async with get_session() as session:
+            rows = await session.scalars(
+                select(CostExcludedExperimentModel).order_by(
+                    CostExcludedExperimentModel.created_at.desc()
+                )
+            )
+            return [_response(row) for row in rows]
+    except ProgrammingError as exc:
+        raise unavailable(exc)
+
+
+@router.post("", response_model=CostExcludedExperimentResponse)
+async def add_cost_excluded_experiment(
+    request: CreateCostExcludedExperimentRequest,
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> CostExcludedExperimentResponse:
+    require_operator_org(auth)
+
+    ref = request.experiment.strip()
+    if not ref:
+        raise HTTPException(status_code=400, detail="experiment must not be empty")
+
+    try:
+        async with get_session() as session:
+            experiment = await _resolve_experiment(session, ref)
+            if getattr(experiment, "is_collection", False):
+                raise HTTPException(
+                    status_code=400,
+                    detail=(
+                        "that experiment is a collection: it homes no trials of "
+                        "its own, so excluding it would exclude nothing. Exclude "
+                        "the experiments that actually ran the work."
+                    ),
+                )
+            existing = await session.scalars(
+                select(CostExcludedExperimentModel).where(
+                    CostExcludedExperimentModel.experiment_id == experiment.id
+                )
+            )
+            if existing.first() is not None:
+                raise HTTPException(
+                    status_code=409, detail="experiment is already excluded"
+                )
+
+            row = CostExcludedExperimentModel(
+                experiment_id=experiment.id,
+                experiment_name=experiment.name,
+                label=request.label.strip(),
+                created_by_user_id=auth.user_id,
+            )
+            session.add(row)
+            try:
+                await session.commit()
+            except IntegrityError:
+                raise HTTPException(
+                    status_code=409, detail="experiment is already excluded"
+                )
+            return _response(row)
+    except ProgrammingError as exc:
+        raise unavailable(exc)
+
+
+@router.delete("/{row_id}")
+async def remove_cost_excluded_experiment(
+    row_id: str,
+    auth: Annotated[AuthContext, Depends(require_admin)],
+) -> dict:
+    require_operator_org(auth)
+    try:
+        async with get_session() as session:
+            await soft_delete(
+                session,
+                CostExcludedExperimentModel,
+                row_id,
+                "cost-excluded experiment not found",
+            )
+    except ProgrammingError as exc:
+        raise unavailable(exc)
+    return {"deleted": row_id}