feat(cost,budgets): impute token cost for subscription runs and enforce it - #11104
Open
trelmitt wants to merge 2 commits into
Open
feat(cost,budgets): impute token cost for subscription runs and enforce it#11104trelmitt wants to merge 2 commits into
trelmitt wants to merge 2 commits into
Conversation
…ce it Subscription runs (e.g. Claude Max) report cost_cents=0, so every cost view and budget metric read zero even while millions of tokens were consumed. This makes that usage visible and governable. - add a calibratable per-model price map + imputeCostCents/effectiveCostCents helpers in @paperclipai/shared (one source of truth for imputation) - surface effectiveCostCents in the cost views (summary, byAgent, byAgentModel, byProject) via a SQL expression generated from that same map (no migration) - add an effective_cents budget metric; computeObservedAmount, evaluateCostEvent, and all three getInvocationBlock scopes enforce it, so a $0 subscription run trips the same soft/hard budget gates as billed spend - tests: model-pricing math, real-SQL subscription imputation, and an effective_cents hard-stop pausing an agent + blocking new work Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
✅ All checks passing — ready for Greptile review and maintainer approval. — commitperclip |
Author
|
Done — searched the PR list and found the related work: linked #3330 and #4430 (subscription-cost imputation) and #10871 (subscription token-usage dashboard) under Related pull requests in the description, and checked the dedup-search box. This PR's distinct piece is the 🤖 Addressed by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
Linked Issues or Issue Description
No public GitHub issue exists. The change is described below with the feature template fields.
Related pull requests
I searched the PR list and found related work. Refs #3330 (open) and #4430 (closed) — both impute an API-equivalent cost for subscription-included runs; the imputation approach overlaps. Refs #10871 (open) — reports token usage where subscription billing zeroes spend; complementary. This PR's distinct piece is the
effective_centsbudget metric that enforces on the imputed cost, so soft and hard budget gates pause a scope on subscription usage — which the above PRs do not do. Maintainers may prefer to consolidate the imputation half with #3330.Subsystem affected
Cost accounting and budgets (
server/src/services/costs.ts,server/src/services/budgets.ts,@paperclipai/shared).Problem or motivation
Subscription runs report
cost_cents = 0. Every cost view and budget metric reads zero even while the runs consume many tokens. A billed-cents hard-stop can never pause a subscription workload, so spend is ungoverned.Proposed solution
Add a calibratable per-model price map with
imputeCostCentsandeffectiveCostCentshelpers. SurfaceeffectiveCostCentsin the cost views. Add aneffective_centsbudget metric and enforce it incomputeObservedAmount,evaluateCostEvent, andgetInvocationBlock.Alternatives considered
Store an imputed cost column at write time. This was rejected because it needs a migration and freezes prices in old rows. Imputing at query time keeps prices editable and avoids schema churn.
Additional context
billed_centsbehavior is unchanged and stays the default metric. The price table holds documented default rates that an operator can tune as provider pricing changes.What Changed
packages/shared/src/model-pricing.ts: a per-model price map plusimputeCostCents()andeffectiveCostCents()(one source of truth for imputation). Exported from the package index.server/src/services/cost-imputation-sql.ts: a SQL sum expression generated from the same price map, so the views and budgets share one definition.server/src/services/costs.ts: reporteffectiveCostCentsinsummary,byAgent,byAgentModel, andbyProject. No schema change.effective_centsvalue toBUDGET_METRICS. UpdatecomputeObservedAmount,evaluateCostEvent, and all threegetInvocationBlockscopes to enforce it. The three per-scope lookups now route through onefirstExceededHardStopPolicyhelper.effective_centshard-stop that pauses an agent and blocks new work.Verification
pnpm --filter @paperclipai/shared build— passes.pnpm --filter @paperclipai/server typecheck— passes.npx vitest run packages/shared/src/model-pricing.test.ts server/src/__tests__/costs-service.test.ts server/src/__tests__/budgets-service.test.ts— 32 tests pass.cost_cents = 0returnseffectiveCostCents = 9150frombyAgentandsummary, and trips a 500-centeffective_centsagent hard-stop (agent paused, hard incident raised, new work blocked).Risks
billed_centsis unchanged and remains the default, so existing policies behave exactly as before.effective_centsis opt-in per policy. No migration runs.Model Used
Claude Opus 4.8 (
claude-opus-4-8), Anthropic. Extended thinking, tool use, and code execution, run through an agentic CLI harness. The model read the code, wrote the change, and ran the tests and typecheck locally.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template#NNN/github.com/paperclipai/paperclipURLs)docs/...,fix/...) and contains no internal Paperclip ticket id or instance-derived details🤖 Generated with Claude Code