Skip to content

feat(server): instance-wide AI stop and daily spend alarm #1367

Description

@FelixTJDietrich

Priority after #1400/#1393 (2026-08-03): no longer blocks #1356 or the first bounded pilot. #1400 already supplies unified metering, refused-unpriced models, separate host/workspace budgets, and pause-on-breach. Use deliberately small budgets during shadow/pilot. The remaining global AI stop and spend push alert become broad-rollout operations hardening.

Problem

A production instance runs detection over all real activity and serves mentor conversations — the two ways it can quietly spend money. Most of the guard now exists (see below); what remains is the part that tells the operator and the part that stops everything at once.

Already shipped (#1400)

  • One usage ledger across detection, replay and mentor (agent/usage/).
  • Two budgets that never mix (host-funded vs workspace-own-provider), enforced during work rather than only after it, with pause-on-breach and resume without restart.
  • Unpriced/unknown models are refused rather than silently billed as zero (LlmUnpricedUsageBlockedException) — stronger than the alert this issue originally asked for.
  • AGENT_ENABLED defaults to false, so a worker started with the documented minimum environment does not claim jobs.

Remaining scope

  • Instance-wide runtime AI stop. Setting the shared budget to zero pauses host-funded work, but a workspace on its own provider keeps running. An instance admin needs one control that halts all LLM work regardless of funding source, flippable without a redeploy, and named in the kill-switch map (docs: production operations runbook #1371).
  • Daily spend alarm. The ledger makes spend visible on demand; nothing pushes. A daily total over a configured threshold should reach the operator through the alerting channel (feat(server): operator alerting for silent failures #1369) — including a breakdown by funding source so "who is spending" is answerable without opening the UI.

Non-goals

Per-user mentor quotas (v2 — the workspace budget is the backstop); hard reservation semantics (the cap is eventually consistent by design, see ADR 0026).

Dependencies

Alarm delivery rides on #1369.

Part of #1354.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    High

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions