Skip to content

feat(admin): add experiment-level cost exclusions - #1132

Open
charlesyhuang wants to merge 6 commits into
stagingfrom
feat/cost-excluded-experiments
Open

feat(admin): add experiment-level cost exclusions#1132
charlesyhuang wants to merge 6 commits into
stagingfrom
feat/cost-excluded-experiments

Conversation

@charlesyhuang

@charlesyhuang charlesyhuang commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Extends cost-exclusion handling to experiments, allowing admins to register experiments (by name or id) whose trials are excluded from cost accounting, matching the existing LLM API key cost-exclusion feature. Implementation adds a new cost_excluded_experiments table and folds a not_excluded_experiment_filter into the shared cost-basis and quota-reservation predicates. The admin dashboard gains a Cost-excluded experiments card grouped with the existing Cost-excluded LLM keys card, backed by a mirrored /admin/cost-excluded-experiments API.

🤖 Generated with Claude Code


Note

Medium Risk
Changes shared quota and cost-basis predicates used for billing enforcement; mistakes could mis-attribute spend, though behavior is covered by router and DB integration tests and mirrors the existing key-exclusion pattern.

Overview
Adds experiment-level cost exclusions alongside the existing LLM-key list: operators can register experiments (by id or unambiguous name) whose home trials are omitted from shared cost accounting.

Data & accounting: New cost_excluded_experiments table (soft-delete, one live row per experiment) and not_excluded_experiment_filter() wired into first_party_spend_filter() and quota inflight predicates so excluded spend does not appear on admin cost dashboards or count toward caps/reservations; removing an entry restores spend. Experiment pages still show raw trial cost.

API & UI: Operator-only GET/POST/DELETE /admin/cost-excluded-experiments, Next.js proxies, and a Cost-excluded experiments card on the admin Costs tab next to cost-excluded keys. Docs note this as a deployment-wide operator control.

Reviewed by Cursor Bugbot for commit 85a81e8. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
oddish-app Ready Ready Preview Aug 8, 2026 11:55pm

Comment thread frontend/src/components/cost-excluded-experiments-card.tsx

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Left a non-blocking comment — Cursor Bugbot reported an unresolved finding, and this PR changes cost/quota accounting with a migration, so it needs human review. Assigned reviewers stateofkate and RishiDesai.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: high. Left a non-blocking comment — Cursor Bugbot on the latest commit completed as skipped, and this PR changes cost/quota accounting with a migration, so it needs human review. Reviewers are already assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

charlesyhuang and others added 4 commits August 8, 2026 15:57
Admin-managed list of experiments whose trials' spend is ignored by cost
accounting, the experiment-level sibling of the cost-excluded LLM keys:
new cost_excluded_experiments table + migration, a
not_excluded_experiment_filter folded into first_party_spend_filter and
the quota inflight predicates, an /admin/cost-excluded-experiments
router, and an admin Costs-tab card grouped with the keys card.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Oddish preview

Commit: 85a81e8416310ce6f47ce39fadccc9385a9af5b5

Surface Link Target
Frontend https://pr-1132.oddish.app Vercel preview for 85a81e8
Backend oddish-pr-1132 oddish-pr-1132
Database project rshdtcxwbjyisteojife project rshdtcxwbjyisteojife

Vercel deployment URL: https://oddish-odrwiupm5.oddish.app

Plan:

  • Frontend deploy: true
  • Backend deploy: false
  • Migrations: false

This comment is updated by the PR Preview workflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 85a81e8. Configure here.

TrialModel.superseded_by_trial_id.is_(None),
TrialModel.status.in_(_INFLIGHT_TRIAL_STATUSES),
not_excluded_llm_key_filter(),
not_excluded_experiment_filter(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quota cancel skips experiment exclusion

High Severity

_active_trial_predicates still only applies not_excluded_llm_key_filter, so trials in cost-excluded experiments are cancelled when a quota fires even though their spend is omitted from settled sums and inflight reservation. That breaks parity with excluded LLM keys, which this path already preserves.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 85a81e8. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants