feat(server): operator alerting for silent failures

[FelixTJDietrich]

Priority

P1 — broad/unattended-rollout gate. Part of #1354; not a blocker for an operator-owned shadow run.

Outcome

Silent operational failures find the operator through one reliable, testable alert contract, and failure of the alert channel itself is visible rather than interpreted as a healthy system.

Alert sources

• dead-letter/poison growth;
• repeated agent-job failures and stalled sync/liveness;
• daily spend threshold and global AI-stop events (feat(server): instance-wide AI stop and daily spend alarm #1367);
• expired/revoked provider credentials and repeated webhook/auth failures;
• backup and restore-rehearsal failures (chore(infra): off-host database backups with a rehearsed restore #1370);
• actionable security/update state for the running release (security(release): make supply-chain evidence complete and enforceable #1546/feat(server,webapp): surface running version and update availability to instance admins #1389).

Delivery contract

• One OperatorAlertService targeting a configured webhook and/or email; avoid a provider matrix.
• Throttle and deduplicate by stable incident identity while preserving occurrence/last-seen counts.
• Send resolved/recovery notification for stateful incidents.
• Provide an admin-authorized test notification.
• Expose last attempt, last success, last failure/reason, and channel state to feat(server,webapp): add an actionable instance Operations overview #1387/feat(server,webapp): guide the first operator from installation to a safe shadow run #1547/feat(server,webapp): identify the running instance and export safe support diagnostics #1548 without exposing the destination secret.
• Every alert includes version/runtime context, impact, observed time, and an exact Operations/runbook link.
• Alert-delivery failure is itself logged/metriced and visible in Operations; do not recursively alert forever through the broken channel.

Done when

• Every alert source has planted trigger, deduplication, throttle, and recovery tests.
• A test alert proves the configured path end to end.
• A broken/revoked destination becomes visible with last-known success and actionable reason.
• Retry behavior is bounded and does not create an alert storm after recovery.
• Cross-tenant alerts contain the minimum operational context and no secrets/raw payloads.
• docs: production operations runbook #1371 documents ownership and the manual fallback when the channel is down.

Non-goals

On-call schedules, escalation policies, incident management, raw-log attachment, a large notification-provider marketplace, or per-event checkbox sprawl.