obs(quotas): measure overshoot cancelled by the enforcement sweep

[kyle-compute]

Why

#1118 removed the blocking advisory lock from trial admission on purpose: the old lock starved workers during over-quota storms. The tradeoff is that admission now checks headroom with no lock, so two concurrent submissions can both observe headroom and both get admitted — briefly overshooting the cap — and the enforcement sweep (cancel_trials_if_quota_reached) claws the overage back afterward.

Whether that overshoot actually happens, how often, and what it costs is currently invisible. Before anyone builds an atomic reservation system (~150-300 LOC), this adds the metric that makes that decision data-driven.

What

One INFO line per enforcement action, emitted inside cancel_trials_if_quota_reached just before the cancellation mutation loop (it reads each trial's pre-cancellation status), computed entirely from the trial rows the sweep already loaded — no new queries, no schema change, no behavior change:

metric=quota.overshoot_cancelled scope=user org_id=org-42 billed_user_id=user-7 trials=3 queued=2 running=1 age_min_s=5.0 age_median_s=30.0 age_max_s=600.0 reserved_usd=2.25

• trials — size of the cancelled set
• queued / running — pre-cancellation status split: queued counts PENDING+QUEUED (never started running — the pure overshoot admissions), running counts RUNNING+RETRYING (crossed the cap mid-flight)
• age_min_s / age_median_s / age_max_s — seconds between each cancelled trial's created_at and now. A cancelled trial that is still QUEUED seconds after creation is the overshoot signature: low ages with a high queued count mean admissions raced past the cap; high ages with running mean spend legitimately crossed the limit mid-flight.
• reserved_usd — per-trial max(cost_usd or 0, pending_trial_reservation_usd), the exact reservation basis _sum_inflight_reserved_usd uses, so the line reads as "how many reserved dollars the sweep released".

Format follows the existing metric=<dotted.name> key=value convention (quota.lock_busy, quota.trials_cancelled, quota.would_block) with lazy %s logging. INFO level: this fires rarely and is the whole point of the change (the sibling quota.trials_cancelled WARNING remains the alerting line; this one carries the analysis fields).

Tests

• test_overshoot_metric_reports_statuses_ages_and_reserved (new, DB-free): three in-memory trials with known statuses/ages/costs; asserts the exact emitted line, including the median and the reserved-USD floor.
• test_org_quota_cancels_every_users_active_trials (extended, DB-backed): asserts the sweep emits the line with trials=4 queued=2 running=2 for the org-scope cancellation it already exercises.

Ran locally against a throwaway Postgres with the oddish alembic schema (plus the backend-owned quotas/org_quotas/quota_bumps tables): tests/test_quota_enforcement.py, tests/test_quota_admission.py, tests/test_quota_lock_try_acquire.py, tests/test_quota_unattributed_retry.py, tests/test_quotas_usage.py, tests/test_quota_config_defaults.py — all pass. ruff check, ruff format, black, and mypy are clean on the changed files.

Note on cost: per-trial reserved cost IS derivable from the loaded rows (cost_usd + the settings reservation floor), so reserved_usd is included rather than skipped.

https://claude.ai/code/session_01MzJSdrHKEqJrJHU7RRJ758

---

Note

Low Risk
Observability-only: one INFO log line before existing cancellation logic; no schema, queries, or enforcement behavior changes.

Overview
Adds metric=quota.overshoot_cancelled INFO logging when the quota enforcement sweep is about to cancel trials, so lock-free admission overshoot can be measured without changing cancellation behavior.

Each line is emitted from cancel_trials_if_quota_reached immediately before the mutation loop, using only the trial rows already loaded: cancelled set size, queued vs running (pre-status), age_min/median/max since created_at, and reserved_usd aligned with inflight reservation (max(cost_usd, pending_trial_reservation_usd)).

Tests cover the exact log format via a DB-free helper test and assert the metric appears during the existing org-scope cancellation integration test.

^{Reviewed by Cursor Bugbot for commit 0629076. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 9, 2026 12:45am

[cursor]

Risk: low. Approved — Cursor Bugbot passed with no findings requiring human review; Cursor Security Agent was not present on this PR. No reviewers assigned.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}

[github-actions]

Oddish preview

Commit: 06290762c734d22e0c8716505ed50a40c60eb59c

Surface	Link	Target
Frontend	https://pr-1136.oddish.app	Vercel preview for 0629076
Backend	oddish-pr-1136	oddish-pr-1136
Database	project orwlteokhqkqqwnrmres	project orwlteokhqkqqwnrmres

Vercel deployment URL: https://oddish-auqqagn4j.oddish.app

Plan:

• Frontend deploy: true
• Backend deploy: true
• Migrations: false

This comment is updated by the PR Preview workflow.