feat: enforce retry budget across outbound calls

[Obaara293]

Overview

This PR adds a sliding-window Retry Budget that limits retries to 10% of successful requests (configurable). Once the budget is exhausted, further retries fail fast to prevent thundering-herd storms against downstream services.

Related Issue

Closes #424

Note

Re-raise of #561, which was closed with an incorrect “not assigned” reason. @Obaara293 was assigned to #424 on Jul 27, 19:57 UTC (~23h before the original PR).

Changes

Retry Budget Engine

• [ADD] internal/resilience/retry_budget.go

  • Ring-buffer implementation with configurable ratio, window size, and bucket count
  • RecordSuccess() / AllowRetry() / RecordRetry() / Available() / Snapshot() API
  • Lazy initialization so zero-value RetryBudget gets sensible defaults
  • Thread-safe; tested with concurrent access

• [MODIFY] internal/outbox/dispatcher.go

  • Added RetryBudget field to DispatcherConfig
  • Integrated budget checks into both drainOnceForPublisher (per-publisher retry loop) and handlePublishError (per-event retry path)
  • Denied retries fail fast with "retry budget exhausted" error message
  • Successes recorded in budget to maintain accurate ratio

• [MODIFY] internal/outbox/metrics.go

  • Added retry_budget_available gauge (0.0–1.0)
  • Added retry_denied_total counter

• [ADD] docs/ops/retry-budget.md

  • Tuning defaults, metrics reference, and alert thresholds

Verification Results

go test ./internal/resilience/...
✅ 12/12 passed

Acceptance Criteria	Status
Budget may not exceed 10% retry ratio over sliding window	✅ Verified with threshold tests
Exhausted budget fails fast (503)	✅ Denied retries fail fast with clear error
retry_budget_available and retry_denied_total metrics emitted	✅ Both registered in Prometheus
Concurrent access is safe	✅ Verified with parallel goroutine test
Documentation in docs/ops/	✅ docs/ops/retry-budget.md

[Obaara293]

@thlpkee20-wq — requesting review of this work.

This is a re-raise of #561, which was closed as “not assigned.” That reason was incorrect:

• I was assigned to Add retry budget to prevent thundering-herd storms against downstreams #424 on Jul 27, 19:57 UTC (by you), with drips-wave acceptance at the same time
• feat: enforce retry budget across outbound calls #561 was opened Jul 28, 18:44 UTC (~23h after assignment)
• Add retry budget to prevent thundering-herd storms against downstreams #424 is still open and still assigned to me

GitHub blocked reopening #561 after the maintainer close, so this PR (#726) was opened from the same branch and links Closes #424. Conflicts with main are now resolved.

Please review when you can. Thanks!